[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:24 dist-test-slave-3kk6 krb5kdc[25106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:24 dist-test-slave-3kk6 krb5kdc[25106](info): set up 2 sockets
Aug 06 02:50:24 dist-test-slave-3kk6 krb5kdc[25106](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:50:25.538123 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:25 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:25.674363 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43153
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:25.803320 25123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:25.803633 25123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:25.803675 25123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:25.807829 25123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:25.807911 25123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:25.807932 25123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:25.807950 25123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:25.807965 25123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:25.813584 25123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:25.814836 25123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:25.815969 25123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:25.825819 25130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:25.825860 25129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:25.825827 25132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:25.825898 25123 server_base.cc:1034] running on GCE node
I20260806 02:50:25.826752 25123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:25.827935 25123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:25.829120 25123 hybrid_clock.cc:648] HybridClock initialized: now 1785984625829137 us; error 74 us; skew 500 ppm
Aug 06 02:50:25 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:25.832185 25123 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:25.833115 25123 webserver.cc:460] Webserver started at http://127.24.130.190:37549/ using document root <none> and password file <none>
I20260806 02:50:25.833698 25123 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:25.833788 25123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:25.834039 25123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:25.836335 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "8ed071ea1c8445159f7c4a06449d63ca"
format_stamp: "Formatted at 2026-08-06 02:50:25 on dist-test-slave-3kk6"
server_key: "a18708bd2ebb3c42020adae038d9beef"
server_key_iv: "acacf4da98ea0b7148f239cbf6f2dc43"
server_key_version: "encryptionkey@0"
I20260806 02:50:25.836956 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "8ed071ea1c8445159f7c4a06449d63ca"
format_stamp: "Formatted at 2026-08-06 02:50:25 on dist-test-slave-3kk6"
server_key: "a18708bd2ebb3c42020adae038d9beef"
server_key_iv: "acacf4da98ea0b7148f239cbf6f2dc43"
server_key_version: "encryptionkey@0"
I20260806 02:50:25.840891 25123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 02:50:25.843467 25141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:25.844789 25123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:50:25.844935 25123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "8ed071ea1c8445159f7c4a06449d63ca"
format_stamp: "Formatted at 2026-08-06 02:50:25 on dist-test-slave-3kk6"
server_key: "a18708bd2ebb3c42020adae038d9beef"
server_key_iv: "acacf4da98ea0b7148f239cbf6f2dc43"
server_key_version: "encryptionkey@0"
I20260806 02:50:25.845062 25123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:25.867929 25123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:25.871381 25123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:25.871603 25123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:25.879292 25123 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43153
I20260806 02:50:25.879310 25216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43153 every 8 connection(s)
I20260806 02:50:25.880326 25123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:25.883227 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 25123
I20260806 02:50:25.883319 25217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:25.883394 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:25.884706 25098 external_mini_cluster.cc:1428] Setting key 8bad2297049116682820f0ca12f394c5
I20260806 02:50:25.890043 25217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca: Bootstrap starting.
I20260806 02:50:25.892428 25217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:25.893226 25217 log.cc:824] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:25.895299 25217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca: No bootstrap required, opened a new log
I20260806 02:50:25.898891 25217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed071ea1c8445159f7c4a06449d63ca" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43153 } }
I20260806 02:50:25.899152 25217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:25.899247 25217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed071ea1c8445159f7c4a06449d63ca, State: Initialized, Role: FOLLOWER
I20260806 02:50:25.899732 25217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed071ea1c8445159f7c4a06449d63ca" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43153 } }
I20260806 02:50:25.899935 25217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:25.900013 25217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:25.900133 25217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:25.901187 25217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed071ea1c8445159f7c4a06449d63ca" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43153 } }
I20260806 02:50:25.901548 25217 leader_election.cc:302] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed071ea1c8445159f7c4a06449d63ca; no voters:
I20260806 02:50:25.901922 25217 leader_election.cc:288] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:25.902022 25222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 1 FOLLOWER]: Leader election won for term 1
Aug 06 02:50:25 dist-test-slave-3kk6 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:25.902256 25222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [term 1 LEADER]: Becoming Leader. State: Replica: 8ed071ea1c8445159f7c4a06449d63ca, State: Running, Role: LEADER
I20260806 02:50:25.902622 25222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed071ea1c8445159f7c4a06449d63ca" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43153 } }
I20260806 02:50:25.903237 25217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:25.903964 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed071ea1c8445159f7c4a06449d63ca. Latest consensus state: current_term: 1 leader_uuid: "8ed071ea1c8445159f7c4a06449d63ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed071ea1c8445159f7c4a06449d63ca" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43153 } } }
I20260806 02:50:25.904114 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:25.904856 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed071ea1c8445159f7c4a06449d63ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed071ea1c8445159f7c4a06449d63ca" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43153 } } }
I20260806 02:50:25.904976 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:25.908794 25244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:25.908864 25244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:25.909569 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.887918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.24.130.190:43153)
0806 02:50:25.888365 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:25.888382 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:25.889519 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 02:50:25.892841 (+ 3322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:25.892858 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:25.893171 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:25.893516 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:25.895844 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.896824 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.898007 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.898471 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.898682 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:25.903310 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:25.903317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:25.903335 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:25.903381 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:25.905706 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.906418 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.906424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.906430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.906582 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.907028 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.907032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.907033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.907552 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:25.907760 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:25.908452 (+ 692us) server_negotiation.cc:299] Negotiation successful
0806 02:50:25.908749 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":160,"threads_started":1}
I20260806 02:50:25.909847 25231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:25.910866 25231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:25.917119 25231 catalog_manager.cc:1347] Generated new cluster ID: c3ebc70ed77b47eda711091b5542a71c
I20260806 02:50:25.917199 25231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:25.940750 25231 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:25.941875 25231 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:25.947988 25231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ed071ea1c8445159f7c4a06449d63ca: Generated new TSK 0
I20260806 02:50:25.948932 25231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:25.991469 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43153
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:26.113574 25252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:26.113868 25252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:26.113940 25252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:26.118093 25252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:26.118175 25252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:26.118276 25252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:26.123728 25252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:26.124794 25252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:26.125836 25252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:26.135757 25258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:26.135810 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:26.135972 25252 server_base.cc:1034] running on GCE node
W20260806 02:50:26.135838 25259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:26.136499 25252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:26.137192 25252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:26.138429 25252 hybrid_clock.cc:648] HybridClock initialized: now 1785984626138391 us; error 29 us; skew 500 ppm
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:26.141517 25252 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:26.142381 25252 webserver.cc:460] Webserver started at http://127.24.130.129:36669/ using document root <none> and password file <none>
I20260806 02:50:26.143011 25252 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:26.143080 25252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:26.143250 25252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:26.145341 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "7a4049edf2da47ffb56ec40a715fde68"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "b4e7ee5601d1bb66ef1824ba53fd8e29"
server_key_iv: "eb4953cc26d5b289090ccf7792e1bfcd"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.145953 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "7a4049edf2da47ffb56ec40a715fde68"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "b4e7ee5601d1bb66ef1824ba53fd8e29"
server_key_iv: "eb4953cc26d5b289090ccf7792e1bfcd"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.150000 25252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:26.152463 25271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.153779 25252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:50:26.154022 25252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "7a4049edf2da47ffb56ec40a715fde68"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "b4e7ee5601d1bb66ef1824ba53fd8e29"
server_key_iv: "eb4953cc26d5b289090ccf7792e1bfcd"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.154148 25252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:26.172577 25252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:26.176285 25252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:26.176522 25252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:26.177201 25252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:26.178161 25252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:26.178242 25252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.178315 25252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:26.178359 25252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.188344 25252 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:34677
I20260806 02:50:26.188373 25421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:34677 every 8 connection(s)
I20260806 02:50:26.189344 25252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:26.189750 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 25252
I20260806 02:50:26.189885 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:26.190209 25098 external_mini_cluster.cc:1428] Setting key 9ecdc47c2bfb914cc5320e9079d7a403
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:26.207129 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.191812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:33149 (local address 127.24.130.190:43153)
0806 02:50:26.191967 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:26.191973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:26.193431 (+ 1458us) server_negotiation.cc:407] Connection header received
0806 02:50:26.194952 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:26.194958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:26.195026 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:26.195134 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:26.196685 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.197298 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.198823 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.199095 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.199224 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.202254 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:26.202259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:26.202262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:26.202298 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:26.204167 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.204947 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.204952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.204954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.205051 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.205597 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.205604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.205609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.205925 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:26.206023 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:26.206829 (+ 806us) server_negotiation.cc:299] Negotiation successful
0806 02:50:26.206987 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:50:26.208338 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.192312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43153 (local address 127.24.130.129:33149)
0806 02:50:26.193208 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:26.193305 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:26.194686 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:26.195350 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:26.195362 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:26.195926 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:26.196489 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:26.196506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.197465 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.197469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:26.198629 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:26.198642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.199282 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.199287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:26.199490 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.200206 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:26.200226 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:26.202004 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:26.204320 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:26.204329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:26.204351 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:26.204793 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:26.205213 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:26.205219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:26.205224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:26.205412 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:26.206044 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:26.206053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:26.206472 (+ 419us) client_negotiation.cc:769] Sending connection context
0806 02:50:26.206755 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 02:50:26.207185 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":124,"threads_started":1}
I20260806 02:50:26.209709 25422 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43153
I20260806 02:50:26.209995 25422 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:26.210700 25422 heartbeater.cc:511] Master 127.24.130.190:43153 requested a full tablet report, sending...
I20260806 02:50:26.212471 25168 ts_manager.cc:194] Registered new tserver with Master: 7a4049edf2da47ffb56ec40a715fde68 (127.24.130.129:34677)
I20260806 02:50:26.213642 25168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:33149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:26.225483 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43153
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:26.343425 25427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:26.343743 25427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:26.343827 25427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:26.348104 25427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:26.348196 25427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:26.348289 25427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:26.353830 25427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:26.355053 25427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:26.356139 25427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:26.366741 25427 server_base.cc:1034] running on GCE node
W20260806 02:50:26.366696 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:26.366783 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:26.366678 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:26.367447 25427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:26.368188 25427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:26.369392 25427 hybrid_clock.cc:648] HybridClock initialized: now 1785984626369377 us; error 34 us; skew 500 ppm
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:26.372335 25427 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:26.373222 25427 webserver.cc:460] Webserver started at http://127.24.130.130:39773/ using document root <none> and password file <none>
I20260806 02:50:26.373767 25427 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:26.373831 25427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:26.374053 25427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:26.376281 25427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "fe213e4635f84e48bcc7bd777ed6bb0a"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "0603aa8d0b972391457a73178268c52f"
server_key_iv: "b0cb44d26fbb0ca0af48331ec0f29e03"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.376886 25427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "fe213e4635f84e48bcc7bd777ed6bb0a"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "0603aa8d0b972391457a73178268c52f"
server_key_iv: "b0cb44d26fbb0ca0af48331ec0f29e03"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.381003 25427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:26.383697 25445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.384985 25427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:26.385138 25427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "fe213e4635f84e48bcc7bd777ed6bb0a"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "0603aa8d0b972391457a73178268c52f"
server_key_iv: "b0cb44d26fbb0ca0af48331ec0f29e03"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.385267 25427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:26.412199 25427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:26.415431 25427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:26.415601 25427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:26.416167 25427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:26.417013 25427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:26.417064 25427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.417106 25427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:26.417128 25427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.426384 25427 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:38501
I20260806 02:50:26.426460 25602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:38501 every 8 connection(s)
I20260806 02:50:26.427407 25427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:26.433917 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 25427
I20260806 02:50:26.434017 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:26.434278 25098 external_mini_cluster.cc:1428] Setting key 2c2980a721bd09bb6f50593da842ef05
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:26.442515 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.429448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:52323 (local address 127.24.130.190:43153)
0806 02:50:26.429592 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:26.429596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:26.430739 (+ 1143us) server_negotiation.cc:407] Connection header received
0806 02:50:26.431720 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:26.431724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:26.431786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:26.431917 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:26.433127 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.433688 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.435025 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.435269 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.435390 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.438241 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:26.438245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:26.438248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:26.438280 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:26.439854 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.440548 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.440551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.440553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.440750 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.441215 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.441217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.441219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.441430 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:26.441511 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:26.442261 (+ 750us) server_negotiation.cc:299] Negotiation successful
0806 02:50:26.442386 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:50:26.443924 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.429840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43153 (local address 127.24.130.130:52323)
0806 02:50:26.430544 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:26.430588 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:26.431497 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:26.432096 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:26.432103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:26.432497 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:26.432980 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:26.432995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.433821 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.433829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:26.434895 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:26.434905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.435431 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.435435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:26.435610 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.436273 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:26.436290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:26.438047 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:26.440003 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:26.440012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:26.440034 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:26.440433 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:26.440894 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:26.440900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:26.440905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:26.441113 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:26.441547 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:26.441559 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:26.441961 (+ 402us) client_negotiation.cc:769] Sending connection context
0806 02:50:26.442212 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:50:26.442640 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":120,"threads_started":1}
I20260806 02:50:26.445348 25603 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43153
I20260806 02:50:26.445631 25603 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:26.446344 25603 heartbeater.cc:511] Master 127.24.130.190:43153 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:50:26.447569 25168 ts_manager.cc:194] Registered new tserver with Master: fe213e4635f84e48bcc7bd777ed6bb0a (127.24.130.130:38501)
I20260806 02:50:26.448455 25168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:52323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:26.468215 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43153
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:26.588047 25611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:26.588375 25611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:26.588423 25611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:26.592782 25611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:26.592869 25611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:26.592932 25611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:26.598373 25611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:26.599591 25611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:26.600615 25611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:26.610556 25622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:26.610597 25616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:26.610579 25611 server_base.cc:1034] running on GCE node
W20260806 02:50:26.610599 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:26.611353 25611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:26.612044 25611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:26.613214 25611 hybrid_clock.cc:648] HybridClock initialized: now 1785984626613181 us; error 42 us; skew 500 ppm
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:26.616119 25611 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:26.616961 25611 webserver.cc:460] Webserver started at http://127.24.130.131:32873/ using document root <none> and password file <none>
I20260806 02:50:26.617480 25611 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:26.617538 25611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:26.617702 25611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:26.619845 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "fdf7eaa94a2a4ff882bac0d091f69967"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "a47f5d4542095edcb4420a6c65fc8341"
server_key_iv: "42e269459e6ac3f852f2c329f85df771"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.620391 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "fdf7eaa94a2a4ff882bac0d091f69967"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "a47f5d4542095edcb4420a6c65fc8341"
server_key_iv: "42e269459e6ac3f852f2c329f85df771"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.624198 25611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:26.626722 25633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.627952 25611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:26.628084 25611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "fdf7eaa94a2a4ff882bac0d091f69967"
format_stamp: "Formatted at 2026-08-06 02:50:26 on dist-test-slave-3kk6"
server_key: "a47f5d4542095edcb4420a6c65fc8341"
server_key_iv: "42e269459e6ac3f852f2c329f85df771"
server_key_version: "encryptionkey@0"
I20260806 02:50:26.628203 25611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:26.639621 25611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:26.642493 25611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:26.642762 25611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:26.643364 25611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:26.644256 25611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:26.644327 25611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.644395 25611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:26.644444 25611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:26.654397 25611 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:33721
I20260806 02:50:26.654417 25790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:33721 every 8 connection(s)
I20260806 02:50:26.655444 25611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:26.656200 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 25611
I20260806 02:50:26.656316 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:26.656602 25098 external_mini_cluster.cc:1428] Setting key 8e55776f682374f69e6820464fd6a96b
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:26.669605 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.657533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:48061 (local address 127.24.130.190:43153)
0806 02:50:26.657751 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:26.657757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:26.659041 (+ 1284us) server_negotiation.cc:407] Connection header received
0806 02:50:26.660050 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:26.660053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:26.660100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:26.660171 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:26.661362 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.661934 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.663226 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.663456 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.663540 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.666066 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:26.666070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:26.666071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:26.666096 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:26.667532 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.668097 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.668105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.668107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.668198 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.668582 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.668585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.668586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.668777 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:26.668856 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:26.669369 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 02:50:26.669475 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260806 02:50:26.670818 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.658008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43153 (local address 127.24.130.131:48061)
0806 02:50:26.658849 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:26.658910 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:26.659847 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:26.660312 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:26.660318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:26.660693 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:26.661173 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:26.661187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.662066 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.662068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:26.663092 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:26.663102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.663595 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.663597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:26.663735 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.664337 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:26.664353 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:26.665903 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:26.667655 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:26.667660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:26.667671 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:26.667969 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:26.668322 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:26.668325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:26.668326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:26.668486 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:26.668894 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:26.668899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:26.669135 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 02:50:26.669316 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:50:26.669582 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":160,"threads_started":1}
I20260806 02:50:26.671936 25791 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43153
I20260806 02:50:26.672166 25791 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:26.672667 25791 heartbeater.cc:511] Master 127.24.130.190:43153 requested a full tablet report, sending...
I20260806 02:50:26.673674 25168 ts_manager.cc:194] Registered new tserver with Master: fdf7eaa94a2a4ff882bac0d091f69967 (127.24.130.131:33721)
I20260806 02:50:26.674257 25168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:48061
I20260806 02:50:26.683423 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:26.712407 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.702387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.24.130.129:34677)
0806 02:50:26.702795 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:26.702819 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:26.702846 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:50:26.702906 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:26.702912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:26.703116 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:26.703245 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:26.704055 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.704924 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.705788 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.706172 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.706261 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.708849 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:26.708860 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:26.708874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:26.708899 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:26.710697 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.711111 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.711116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.711118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.711203 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.711444 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.711446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.711448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.711709 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:26.711880 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:26.712027 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:50:26.712189 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
W20260806 02:50:26.713541 25308 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55748
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:26.742383 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:26.733836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.24.130.129:34677)
0806 02:50:26.734025 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:26.734029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:26.734102 (+ 73us) server_negotiation.cc:407] Connection header received
0806 02:50:26.734227 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:26.734229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:26.734282 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:26.734369 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:26.735134 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.735650 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.736456 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:26.736673 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:26.736758 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:26.739387 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:26.739390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:26.739392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:26.739417 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:26.740794 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.741250 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.741253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.741254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.741341 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:26.741588 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:26.741591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:26.741592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:26.741754 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:26.741867 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:26.742073 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:50:26.742180 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:50:26.743469 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 25252
I20260806 02:50:26.754968 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 25427
I20260806 02:50:26.773936 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 25611
I20260806 02:50:26.783509 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 25123
2026-08-06T02:50:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2489 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25809](info): set up 2 sockets
Aug 06 02:50:26 dist-test-slave-3kk6 krb5kdc[25809](info): commencing operation
krb5kdc: starting...
W20260806 02:50:28.017808 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:28.155723 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39183
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:28.278328 25825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:28.278734 25825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:28.278867 25825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:28.283106 25825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:28.283190 25825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:28.283209 25825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:28.283228 25825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:28.283281 25825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:28.289006 25825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:28.290246 25825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:28.291409 25825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:28.300676 25833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.300676 25832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.300673 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:28.301084 25825 server_base.cc:1034] running on GCE node
I20260806 02:50:28.301743 25825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:28.303093 25825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:28.304257 25825 hybrid_clock.cc:648] HybridClock initialized: now 1785984628304227 us; error 50 us; skew 500 ppm
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:28.307430 25825 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:28.308390 25825 webserver.cc:460] Webserver started at http://127.24.130.190:38225/ using document root <none> and password file <none>
I20260806 02:50:28.308964 25825 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:28.309029 25825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:28.309212 25825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:28.311499 25825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "b96359c21da526ac1b7d29e83332fea2"
server_key_iv: "3c6e3ba883eb3beb138013ea5922ecbf"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.312152 25825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "b96359c21da526ac1b7d29e83332fea2"
server_key_iv: "3c6e3ba883eb3beb138013ea5922ecbf"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.316076 25825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:28.318544 25849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.320020 25825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:28.320134 25825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "b96359c21da526ac1b7d29e83332fea2"
server_key_iv: "3c6e3ba883eb3beb138013ea5922ecbf"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.320232 25825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:28.350483 25825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:28.354429 25825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:28.354645 25825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:28.362317 25825 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:39183
I20260806 02:50:28.362397 25931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:39183 every 8 connection(s)
I20260806 02:50:28.363632 25825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:28.366577 25932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:28.372475 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8: Bootstrap starting.
I20260806 02:50:28.373873 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 25825
I20260806 02:50:28.373974 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:28.374253 25098 external_mini_cluster.cc:1428] Setting key 934973e8378f0c86315703c21918d488
I20260806 02:50:28.375416 25932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:28.376391 25932 log.cc:824] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:28.378835 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8: No bootstrap required, opened a new log
I20260806 02:50:28.383217 25932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39183 } }
I20260806 02:50:28.383476 25932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:28.383512 25932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d35b0e6f4ad40dd9cad78f180d6fbd8, State: Initialized, Role: FOLLOWER
I20260806 02:50:28.384017 25932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39183 } }
I20260806 02:50:28.384179 25932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:28.384269 25932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:28.384402 25932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:28.385653 25932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39183 } }
I20260806 02:50:28.386132 25932 leader_election.cc:302] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d35b0e6f4ad40dd9cad78f180d6fbd8; no voters:
I20260806 02:50:28.386611 25932 leader_election.cc:288] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:28.386798 25938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:28.387099 25938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [term 1 LEADER]: Becoming Leader. State: Replica: 8d35b0e6f4ad40dd9cad78f180d6fbd8, State: Running, Role: LEADER
I20260806 02:50:28.387549 25938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39183 } }
I20260806 02:50:28.388239 25932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:28.389569 25939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39183 } } }
I20260806 02:50:28.389815 25939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:28.390224 25940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d35b0e6f4ad40dd9cad78f180d6fbd8. Latest consensus state: current_term: 1 leader_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d35b0e6f4ad40dd9cad78f180d6fbd8" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39183 } } }
I20260806 02:50:28.390434 25940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:28.391203 25952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:28.392146 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.375523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.24.130.190:39183)
0806 02:50:28.376045 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:28.376066 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:28.376111 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:50:28.377063 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:28.377085 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:28.377525 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:28.377908 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:28.378692 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.380191 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.381182 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.382020 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.382199 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.385417 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:28.385428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:28.385451 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:28.385500 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:28.387831 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.388573 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.388583 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.388592 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.388748 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.389140 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.389144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.389145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.389708 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:28.389947 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:28.390978 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0806 02:50:28.391332 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":162,"threads_started":1}
I20260806 02:50:28.394332 25952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:28.400514 25952 catalog_manager.cc:1347] Generated new cluster ID: 69e2ff14363c46e2a50350d3e097488f
I20260806 02:50:28.400610 25952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:28.424362 25952 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:28.425503 25952 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:28.437014 25952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d35b0e6f4ad40dd9cad78f180d6fbd8: Generated new TSK 0
I20260806 02:50:28.437865 25952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:28.478066 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39183
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:28.601047 25967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:28.601382 25967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:28.601429 25967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:28.605861 25967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:28.605953 25967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:28.606017 25967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:28.611840 25967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:28.613047 25967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:28.614161 25967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:28.624495 25973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.624495 25976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.624506 25974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:28.625167 25967 server_base.cc:1034] running on GCE node
I20260806 02:50:28.625594 25967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:28.626284 25967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:28.627440 25967 hybrid_clock.cc:648] HybridClock initialized: now 1785984628627409 us; error 46 us; skew 500 ppm
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:28.630290 25967 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:28.631327 25967 webserver.cc:460] Webserver started at http://127.24.130.129:46837/ using document root <none> and password file <none>
I20260806 02:50:28.631947 25967 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:28.632045 25967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:28.632305 25967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:28.634474 25967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "6c93eb9a74f64a84824ca723f4f5271b"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "a65eeab309c9065b9107f53b40073306"
server_key_iv: "ffaaaad1468d930dd87f328151c59068"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.635118 25967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "6c93eb9a74f64a84824ca723f4f5271b"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "a65eeab309c9065b9107f53b40073306"
server_key_iv: "ffaaaad1468d930dd87f328151c59068"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.639117 25967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:28.641767 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.643064 25967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:28.643213 25967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "6c93eb9a74f64a84824ca723f4f5271b"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "a65eeab309c9065b9107f53b40073306"
server_key_iv: "ffaaaad1468d930dd87f328151c59068"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.643352 25967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:28.658138 25967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:28.661690 25967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:28.661938 25967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:28.662616 25967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:28.663573 25967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:28.663643 25967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.663715 25967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:28.663761 25967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.673170 25967 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:37319
I20260806 02:50:28.673200 26142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:37319 every 8 connection(s)
I20260806 02:50:28.674172 25967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:28.676327 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 25967
I20260806 02:50:28.676445 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:28.676750 25098 external_mini_cluster.cc:1428] Setting key 8c74c09923e32c71bb2ddf116a2d192c
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:50:28.690578 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.676116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:48939 (local address 127.24.130.190:39183)
0806 02:50:28.676272 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:28.676276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:28.677377 (+ 1101us) server_negotiation.cc:407] Connection header received
0806 02:50:28.678877 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:28.678883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:28.678947 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:28.679051 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:28.680284 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.680846 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.682081 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.682340 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.682451 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.686013 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:28.686017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:28.686021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:28.686055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:28.687958 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.688699 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.688705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.688710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.688819 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.689292 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.689295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.689296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.689549 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:28.689665 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:28.690248 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 02:50:28.690374 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:28.691658 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.676615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39183 (local address 127.24.130.129:48939)
0806 02:50:28.677142 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:28.677222 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:28.678599 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:28.679215 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:28.679223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:28.679608 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:28.680109 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:28.680123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.680992 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.681001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:28.681942 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:28.681952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.682512 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.682516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:28.682748 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.683456 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:28.683484 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:28.685741 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:28.688128 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:28.688134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:28.688150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:28.688578 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:28.689005 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:28.689007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:28.689009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:28.689187 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:28.689671 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:28.689677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:28.690011 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 02:50:28.690189 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 02:50:28.690494 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":170,"threads_started":1}
I20260806 02:50:28.694810 26143 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39183
I20260806 02:50:28.695111 26143 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:28.695811 26143 heartbeater.cc:511] Master 127.24.130.190:39183 requested a full tablet report, sending...
I20260806 02:50:28.697830 25878 ts_manager.cc:194] Registered new tserver with Master: 6c93eb9a74f64a84824ca723f4f5271b (127.24.130.129:37319)
I20260806 02:50:28.699102 25878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:48939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:28.709484 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39183
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:28.835213 26148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:28.835503 26148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:28.835548 26148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:28.839679 26148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:28.839761 26148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:28.839823 26148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:28.845317 26148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:28.846392 26148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:28.847522 26148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:28.857609 26156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.857658 26161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.857864 26158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:28.858083 26148 server_base.cc:1034] running on GCE node
I20260806 02:50:28.858515 26148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:28.859268 26148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:28.860512 26148 hybrid_clock.cc:648] HybridClock initialized: now 1785984628860491 us; error 32 us; skew 500 ppm
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:28.863948 26148 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:28.865096 26148 webserver.cc:460] Webserver started at http://127.24.130.130:40835/ using document root <none> and password file <none>
I20260806 02:50:28.865692 26148 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:28.865770 26148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:28.865971 26148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:28.868327 26148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "7f55e6e13b054ec188a3309419529e6e"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "f99ec4b7f0d7adda2337de1db98f0f58"
server_key_iv: "5ee9b83f3333e30f728f3042273593c1"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.868927 26148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "7f55e6e13b054ec188a3309419529e6e"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "f99ec4b7f0d7adda2337de1db98f0f58"
server_key_iv: "5ee9b83f3333e30f728f3042273593c1"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.873155 26148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:28.876123 26172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.877681 26148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:28.877831 26148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "7f55e6e13b054ec188a3309419529e6e"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-3kk6"
server_key: "f99ec4b7f0d7adda2337de1db98f0f58"
server_key_iv: "5ee9b83f3333e30f728f3042273593c1"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.877957 26148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:28.893988 26148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:28.897850 26148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:28.898034 26148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:28.898712 26148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:28.899641 26148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:28.899693 26148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.899739 26148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:28.899775 26148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.909703 26148 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:41483
I20260806 02:50:28.909785 26322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:41483 every 8 connection(s)
I20260806 02:50:28.910800 26148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:28.918148 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 26148
I20260806 02:50:28.918291 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:28.918586 25098 external_mini_cluster.cc:1428] Setting key d3b4ee9ddafd87f0091df43793a52572
Aug 06 02:50:28 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:28.927838 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.912823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:49289 (local address 127.24.130.190:39183)
0806 02:50:28.913002 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:28.913007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:28.913862 (+ 855us) server_negotiation.cc:407] Connection header received
0806 02:50:28.915320 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:28.915323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:28.915373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:28.915466 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:28.916723 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.917273 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.918916 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.919142 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.919273 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.922727 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:28.922732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:28.922738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:28.922775 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:28.924756 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.925550 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.925555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.925559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.925689 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.926199 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.926205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.926209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.926513 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:28.926644 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:28.927440 (+ 796us) server_negotiation.cc:299] Negotiation successful
0806 02:50:28.927604 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:50:28.929345 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.913219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39183 (local address 127.24.130.130:49289)
0806 02:50:28.913693 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:28.913741 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:28.915101 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:28.915630 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:28.915637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:28.916053 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:28.916561 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:28.916575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.917389 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.917391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:28.918783 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:28.918796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.919336 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.919340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:28.919492 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.920255 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:28.920277 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:28.922454 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:28.924911 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:28.924922 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:28.924942 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:28.925375 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:28.925841 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:28.925847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:28.925852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:28.926076 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:28.926690 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:28.926699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:28.927096 (+ 397us) client_negotiation.cc:769] Sending connection context
0806 02:50:28.927353 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 02:50:28.927767 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:50:28.932360 26324 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39183
I20260806 02:50:28.932639 26324 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:28.933393 26324 heartbeater.cc:511] Master 127.24.130.190:39183 requested a full tablet report, sending...
I20260806 02:50:28.934898 25878 ts_manager.cc:194] Registered new tserver with Master: 7f55e6e13b054ec188a3309419529e6e (127.24.130.130:41483)
I20260806 02:50:28.935835 25878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:49289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:28.959064 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39183
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:29.082463 26330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:29.082888 26330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:29.082942 26330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:29.086973 26330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:29.087045 26330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:29.087108 26330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:29.092885 26330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:29.094025 26330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:29.095146 26330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:29.105485 26330 server_base.cc:1034] running on GCE node
W20260806 02:50:29.105427 26339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:29.105440 26343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:29.105453 26338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:29.106207 26330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:29.106966 26330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:29.108152 26330 hybrid_clock.cc:648] HybridClock initialized: now 1785984629108134 us; error 32 us; skew 500 ppm
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:29.111197 26330 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:29.112193 26330 webserver.cc:460] Webserver started at http://127.24.130.131:45869/ using document root <none> and password file <none>
I20260806 02:50:29.112795 26330 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:29.112891 26330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:29.113137 26330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:29.115401 26330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-3kk6"
server_key: "3be88fb20da77dec846fcbc033bf4946"
server_key_iv: "205e1eae05b609ac12007d12898a52cb"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.116021 26330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-3kk6"
server_key: "3be88fb20da77dec846fcbc033bf4946"
server_key_iv: "205e1eae05b609ac12007d12898a52cb"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.120167 26330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:29.122888 26353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.124315 26330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:29.124461 26330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-3kk6"
server_key: "3be88fb20da77dec846fcbc033bf4946"
server_key_iv: "205e1eae05b609ac12007d12898a52cb"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.124588 26330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:29.137058 26330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:29.140381 26330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:29.140612 26330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:29.141319 26330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:29.142239 26330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:29.142310 26330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.142380 26330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:29.142428 26330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.152066 26330 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:45963
I20260806 02:50:29.152084 26517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:45963 every 8 connection(s)
I20260806 02:50:29.153160 26330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:29.158968 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 26330
I20260806 02:50:29.159109 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:29.159380 25098 external_mini_cluster.cc:1428] Setting key 11c2a598278d57c6ae45e1ea1995636c
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:29.167826 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.155207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:60569 (local address 127.24.130.190:39183)
0806 02:50:29.155361 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.155365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.156180 (+ 815us) server_negotiation.cc:407] Connection header received
0806 02:50:29.157552 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.157556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.157619 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.157720 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.159015 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.159704 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.161293 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.161626 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.161769 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.164295 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.164298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.164300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.164327 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.165643 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.166144 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.166147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.166149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.166252 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.166746 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.166749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.166751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.166950 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.167042 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.167586 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.167695 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:50:29.168720 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.155558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39183 (local address 127.24.130.131:60569)
0806 02:50:29.156021 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.156066 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.157331 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.157905 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.157912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.158295 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:29.158865 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.158881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.159868 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.159872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.161162 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.161172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.161816 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.161819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.161968 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.162553 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:29.162569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:29.164128 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:29.165750 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:29.165755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:29.165768 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:29.166028 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:29.166377 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:29.166381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:29.166383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:29.166551 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:29.167078 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:29.167084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:29.167346 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.167527 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.167811 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260806 02:50:29.169919 26520 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39183
I20260806 02:50:29.170166 26520 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:29.170727 26520 heartbeater.cc:511] Master 127.24.130.190:39183 requested a full tablet report, sending...
I20260806 02:50:29.171895 25878 ts_manager.cc:194] Registered new tserver with Master: 2e22fe4d86bf4785b56d3c1c4ccc9785 (127.24.130.131:45963)
I20260806 02:50:29.172662 25878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:60569
I20260806 02:50:29.173611 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:29.185231 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.177552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.24.130.190:39183)
0806 02:50:29.177697 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.177701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.177906 (+ 205us) server_negotiation.cc:407] Connection header received
0806 02:50:29.177984 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.177986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.178029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.178112 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.178969 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.179488 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.180355 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.180573 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.180667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.182236 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.182239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.182242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.182268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.183699 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.184104 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.184107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.184108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.184186 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.184417 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.184419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.184420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.184635 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.184739 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.184929 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.185028 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:29.190726 25878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:29.193958 25878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:29.210188 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.203478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:45963 (local address 127.0.0.1:50608)
0806 02:50:29.203871 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.203937 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.204244 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.205466 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.205469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.205494 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.205691 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.205700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.207261 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.207266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.208576 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.208586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.209511 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.209515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.209673 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.209721 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.209843 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.209976 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":381,"spinlock_wait_cycles":11392,"thread_start_us":108,"threads_started":1}
I20260806 02:50:29.210183 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.203677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.24.130.131:45963)
0806 02:50:29.205070 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.205081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.205100 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:50:29.205159 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.205163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.205323 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.205586 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.205820 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.207072 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.208704 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.209382 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.209514 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.209650 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.209890 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.209994 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1299,"thread_start_us":187,"threads_started":1}
I20260806 02:50:29.210613 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.203826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:41483 (local address 127.0.0.1:33840)
0806 02:50:29.204575 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.204589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.204683 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.205456 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.205460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.205488 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.205734 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.205752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.207295 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.207299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.209068 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.209079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.210139 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.210142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.210282 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.210331 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.210389 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.210463 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":421,"spinlock_wait_cycles":896,"thread_start_us":113,"threads_started":1}
I20260806 02:50:29.211205 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.204475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.24.130.129:37319)
0806 02:50:29.204963 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.204973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.204998 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:50:29.205077 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.205082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.205265 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.205427 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.206828 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.207723 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.209255 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.209994 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.210720 (+ 726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.210878 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.210937 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.211036 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":100,"threads_started":1}
I20260806 02:50:29.210630 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.204259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:37319 (local address 127.0.0.1:57292)
0806 02:50:29.204724 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.204738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.204834 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.205428 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.205434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.205458 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.205702 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.205708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.207888 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.207891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.209069 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.209079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.210144 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.210148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.210316 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.210352 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.210428 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.210492 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":149,"spinlock_wait_cycles":896,"thread_start_us":92,"threads_started":1}
I20260806 02:50:29.211737 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.204300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.24.130.130:41483)
0806 02:50:29.204955 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.204968 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.204992 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:50:29.205073 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.205079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.205281 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.205904 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.206076 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.207159 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.209209 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.209994 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.210777 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.210943 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.210994 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.211572 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":150,"threads_started":1}
I20260806 02:50:29.213879 26426 tablet_service.cc:1467] Processing CreateTablet for tablet 6cef0cbfc4614023aa0a7d2213a1fb79 (DEFAULT_TABLE table=test-table [id=84038a9adca745a6855e8f63ecc9ced1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:29.214218 26240 tablet_service.cc:1467] Processing CreateTablet for tablet 6cef0cbfc4614023aa0a7d2213a1fb79 (DEFAULT_TABLE table=test-table [id=84038a9adca745a6855e8f63ecc9ced1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:29.214218 26049 tablet_service.cc:1467] Processing CreateTablet for tablet 6cef0cbfc4614023aa0a7d2213a1fb79 (DEFAULT_TABLE table=test-table [id=84038a9adca745a6855e8f63ecc9ced1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:29.215066 26426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cef0cbfc4614023aa0a7d2213a1fb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:29.215263 26240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cef0cbfc4614023aa0a7d2213a1fb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:29.215256 26049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cef0cbfc4614023aa0a7d2213a1fb79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:29.221757 26542 tablet_bootstrap.cc:492] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: Bootstrap starting.
I20260806 02:50:29.222914 26543 tablet_bootstrap.cc:492] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: Bootstrap starting.
I20260806 02:50:29.224133 26542 tablet_bootstrap.cc:654] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:29.224802 26543 tablet_bootstrap.cc:654] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:29.225364 26542 log.cc:824] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:29.225646 26543 log.cc:824] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:29.226212 26544 tablet_bootstrap.cc:492] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: Bootstrap starting.
I20260806 02:50:29.227790 26542 tablet_bootstrap.cc:492] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: No bootstrap required, opened a new log
I20260806 02:50:29.227867 26543 tablet_bootstrap.cc:492] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: No bootstrap required, opened a new log
I20260806 02:50:29.228012 26542 ts_tablet_manager.cc:1397] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:50:29.228061 26543 ts_tablet_manager.cc:1397] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:50:29.229022 26544 tablet_bootstrap.cc:654] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:29.229923 26544 log.cc:824] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:29.231374 26542 raft_consensus.cc:348] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.231645 26542 raft_consensus.cc:374] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:29.231735 26542 raft_consensus.cc:729] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e22fe4d86bf4785b56d3c1c4ccc9785, State: Initialized, Role: FOLLOWER
I20260806 02:50:29.232201 26544 tablet_bootstrap.cc:492] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: No bootstrap required, opened a new log
I20260806 02:50:29.232298 26542 consensus_queue.cc:260] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.232407 26544 ts_tablet_manager.cc:1397] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:50:29.233412 26542 ts_tablet_manager.cc:1428] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:50:29.233355 26543 raft_consensus.cc:348] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.233613 26520 heartbeater.cc:503] Master 127.24.130.190:39183 was elected leader, sending a full tablet report...
I20260806 02:50:29.233705 26543 raft_consensus.cc:374] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:29.233789 26543 raft_consensus.cc:729] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c93eb9a74f64a84824ca723f4f5271b, State: Initialized, Role: FOLLOWER
I20260806 02:50:29.234251 26543 consensus_queue.cc:260] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.235064 26143 heartbeater.cc:503] Master 127.24.130.190:39183 was elected leader, sending a full tablet report...
I20260806 02:50:29.235296 26543 ts_tablet_manager.cc:1428] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:50:29.237569 26544 raft_consensus.cc:348] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.237892 26544 raft_consensus.cc:374] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:29.237960 26544 raft_consensus.cc:729] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f55e6e13b054ec188a3309419529e6e, State: Initialized, Role: FOLLOWER
I20260806 02:50:29.238416 26544 consensus_queue.cc:260] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.239107 26324 heartbeater.cc:503] Master 127.24.130.190:39183 was elected leader, sending a full tablet report...
I20260806 02:50:29.239439 26544 ts_tablet_manager.cc:1428] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 02:50:29.404289 26522 tablet.cc:2369] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:29.412146 26325 tablet.cc:2369] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:29.425662 26144 tablet.cc:2369] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:29.439999 26549 raft_consensus.cc:482] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:29.440225 26549 raft_consensus.cc:504] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.441462 26549 leader_election.cc:288] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e22fe4d86bf4785b56d3c1c4ccc9785 (127.24.130.131:45963), 7f55e6e13b054ec188a3309419529e6e (127.24.130.130:41483)
I20260806 02:50:29.447428 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.441854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:38451 (local address 127.24.130.131:45963)
0806 02:50:29.441996 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.442003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.442300 (+ 297us) server_negotiation.cc:407] Connection header received
0806 02:50:29.442456 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.442459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.442515 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.442602 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.443156 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.443998 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.445984 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.446701 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.446851 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.446912 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.447205 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.447291 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:50:29.447475 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.441884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:43675 (local address 127.24.130.130:41483)
0806 02:50:29.442024 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.442029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.442303 (+ 274us) server_negotiation.cc:407] Connection header received
0806 02:50:29.442454 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.442458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.442519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.442603 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.443156 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.443977 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.445984 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.446701 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.446850 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.446912 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.447268 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.447347 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:29.447535 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.441810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:41483 (local address 127.24.130.129:43675)
0806 02:50:29.442168 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.442182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.442326 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.442687 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.442690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.442754 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.442993 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.443001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.444143 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.444147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.445812 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.445827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.446854 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.446859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.447043 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.447087 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.447228 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.447335 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":222,"spinlock_wait_cycles":99712,"thread_start_us":130,"threads_started":1}
I20260806 02:50:29.447535 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.441708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:45963 (local address 127.24.130.129:38451)
0806 02:50:29.442152 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.442168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.442326 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.442687 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.442690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.442756 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.442993 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.443001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.444143 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.444147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.445812 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.445827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.446854 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.446859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.447055 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.447095 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.447184 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.447281 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":286,"spinlock_wait_cycles":24448,"thread_start_us":151,"threads_started":1}
I20260806 02:50:29.448313 26449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cef0cbfc4614023aa0a7d2213a1fb79" candidate_uuid: "6c93eb9a74f64a84824ca723f4f5271b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" is_pre_election: true
I20260806 02:50:29.448345 26262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cef0cbfc4614023aa0a7d2213a1fb79" candidate_uuid: "6c93eb9a74f64a84824ca723f4f5271b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f55e6e13b054ec188a3309419529e6e" is_pre_election: true
I20260806 02:50:29.448664 26262 raft_consensus.cc:2454] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c93eb9a74f64a84824ca723f4f5271b in term 0.
I20260806 02:50:29.448665 26449 raft_consensus.cc:2454] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c93eb9a74f64a84824ca723f4f5271b in term 0.
I20260806 02:50:29.449213 25985 leader_election.cc:302] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e22fe4d86bf4785b56d3c1c4ccc9785, 6c93eb9a74f64a84824ca723f4f5271b, 7f55e6e13b054ec188a3309419529e6e; no voters:
I20260806 02:50:29.449560 26549 raft_consensus.cc:2781] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:29.449649 26549 raft_consensus.cc:482] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:29.449680 26549 raft_consensus.cc:3037] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:29.450886 26549 raft_consensus.cc:504] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.451310 26549 leader_election.cc:288] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [CANDIDATE]: Term 1 election: Requested vote from peers 2e22fe4d86bf4785b56d3c1c4ccc9785 (127.24.130.131:45963), 7f55e6e13b054ec188a3309419529e6e (127.24.130.130:41483)
I20260806 02:50:29.451706 26449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cef0cbfc4614023aa0a7d2213a1fb79" candidate_uuid: "6c93eb9a74f64a84824ca723f4f5271b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785"
I20260806 02:50:29.451736 26262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cef0cbfc4614023aa0a7d2213a1fb79" candidate_uuid: "6c93eb9a74f64a84824ca723f4f5271b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f55e6e13b054ec188a3309419529e6e"
I20260806 02:50:29.451879 26449 raft_consensus.cc:3037] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:29.451874 26262 raft_consensus.cc:3037] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:29.453001 26449 raft_consensus.cc:2454] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 2e22fe4d86bf4785b56d3c1c4ccc9785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93eb9a74f64a84824ca723f4f5271b in term 1.
I20260806 02:50:29.453015 26262 raft_consensus.cc:2454] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 7f55e6e13b054ec188a3309419529e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c93eb9a74f64a84824ca723f4f5271b in term 1.
I20260806 02:50:29.453372 25988 leader_election.cc:302] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e22fe4d86bf4785b56d3c1c4ccc9785, 6c93eb9a74f64a84824ca723f4f5271b; no voters:
I20260806 02:50:29.453572 26549 raft_consensus.cc:2781] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:29.453846 26549 raft_consensus.cc:686] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [term 1 LEADER]: Becoming Leader. State: Replica: 6c93eb9a74f64a84824ca723f4f5271b, State: Running, Role: LEADER
I20260806 02:50:29.454197 26549 consensus_queue.cc:237] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } }
I20260806 02:50:29.457731 25878 catalog_manager.cc:5495] T 6cef0cbfc4614023aa0a7d2213a1fb79 P 6c93eb9a74f64a84824ca723f4f5271b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c93eb9a74f64a84824ca723f4f5271b (127.24.130.129). New cstate: current_term: 1 leader_uuid: "6c93eb9a74f64a84824ca723f4f5271b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e22fe4d86bf4785b56d3c1c4ccc9785" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c93eb9a74f64a84824ca723f4f5271b" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f55e6e13b054ec188a3309419529e6e" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 41483 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:29.512447 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.497658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.24.130.129:37319)
0806 02:50:29.497837 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.497842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.498017 (+ 175us) server_negotiation.cc:407] Connection header received
0806 02:50:29.498200 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.498206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.498285 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.498377 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.499202 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.499731 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.500641 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.500877 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.500967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.507928 (+ 6961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.507946 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.507964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.508002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.510057 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.510706 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.510713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.510714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.510834 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.511202 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.511205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.511207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.511550 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.511779 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.512057 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.512237 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:29.544302 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.534646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.24.130.129:37319)
0806 02:50:29.534896 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.534900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.534989 (+ 89us) server_negotiation.cc:407] Connection header received
0806 02:50:29.535172 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.535176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.535232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.535319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.536098 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.536651 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.537526 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.537797 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.537898 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.540671 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.540674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.540676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.540707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.542268 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.542856 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.542859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.542860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.542949 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.543269 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.543272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.543273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.543498 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.543635 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.544015 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.544163 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[25809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:29.574872 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.565337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.24.130.129:37319)
0806 02:50:29.565546 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.565553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.565724 (+ 171us) server_negotiation.cc:407] Connection header received
0806 02:50:29.565936 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.565939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.565988 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.566078 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.566885 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.567411 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.568298 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.568528 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.568640 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.571661 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.571664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.571666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.571709 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.573028 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.573497 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.573500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.573502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.573584 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.573875 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.573879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.573881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.574071 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.574228 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.574518 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.574700 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260806 02:50:29.576449 26021 tablet_service.cc:2908] Rejecting scan request for tablet 6cef0cbfc4614023aa0a7d2213a1fb79: Uninitialized: clean time has not yet been initialized
I20260806 02:50:29.578753 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 25967
I20260806 02:50:29.593117 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 26148
I20260806 02:50:29.606560 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 26330
I20260806 02:50:29.630800 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 25825
2026-08-06T02:50:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2850 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[26569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[26569](info): set up 2 sockets
Aug 06 02:50:29 dist-test-slave-3kk6 krb5kdc[26569](info): commencing operation
krb5kdc: starting...
W20260806 02:50:30.870981 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:30 dist-test-slave-3kk6 krb5kdc[26569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:31.008249 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:40325
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:31.128261 26590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:31.128639 26590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:31.128718 26590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:31.132984 26590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:31.133124 26590 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:50:31.133157 26590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:31.133194 26590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:31.133248 26590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:31.133288 26590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:31.139292 26590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:40325
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:31.140591 26590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:31.141753 26590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:31.150839 26598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.150879 26600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.150889 26596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.151130 26590 server_base.cc:1034] running on GCE node
I20260806 02:50:31.151790 26590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:31.152992 26590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:31.154199 26590 hybrid_clock.cc:648] HybridClock initialized: now 1785984631154174 us; error 46 us; skew 500 ppm
Aug 06 02:50:31 dist-test-slave-3kk6 krb5kdc[26569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984631, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:31.157271 26590 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:31.158420 26590 webserver.cc:460] Webserver started at http://127.24.130.190:44097/ using document root <none> and password file <none>
I20260806 02:50:31.159085 26590 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:31.159183 26590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:31.159430 26590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:31.161773 26590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "cf025023d150485ea748b1abb4607c3b"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "a3e824c23be92c5558a22d52b31bd97e"
server_key_iv: "438731f358d34652ca379eb8a685fa04"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.162405 26590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "cf025023d150485ea748b1abb4607c3b"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "a3e824c23be92c5558a22d52b31bd97e"
server_key_iv: "438731f358d34652ca379eb8a685fa04"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.166615 26590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:31.169410 26610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.170964 26590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:31.171125 26590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "cf025023d150485ea748b1abb4607c3b"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "a3e824c23be92c5558a22d52b31bd97e"
server_key_iv: "438731f358d34652ca379eb8a685fa04"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.171262 26590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:31.211061 26590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:31.214885 26590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:31.215075 26590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:31.223424 26590 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:40325
I20260806 02:50:31.223487 26696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:40325 every 8 connection(s)
I20260806 02:50:31.224490 26590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:31.227072 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 26590
I20260806 02:50:31.227171 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:31.227455 25098 external_mini_cluster.cc:1428] Setting key 89c20ee811c3067f728807789931f354
I20260806 02:50:31.227685 26697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:31.234817 26697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: Bootstrap starting.
Aug 06 02:50:31 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:31.237346 26697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:31.238243 26697 log.cc:824] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:31.240500 26697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: No bootstrap required, opened a new log
I20260806 02:50:31.243567 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.229036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.24.130.190:40325)
0806 02:50:31.229564 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.229588 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.229630 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:50:31.230536 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.230557 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.231015 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.231380 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.232287 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.233342 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.234252 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.234831 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.235021 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.237705 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.237711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.237726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:31.237760 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.240427 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.240971 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.240976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.240981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.241108 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.241431 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.241434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.241435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.241860 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.242070 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.242402 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.242776 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":159,"threads_started":1}
I20260806 02:50:31.243894 26697 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf025023d150485ea748b1abb4607c3b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 40325 } }
I20260806 02:50:31.244210 26697 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:31.244278 26697 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf025023d150485ea748b1abb4607c3b, State: Initialized, Role: FOLLOWER
I20260806 02:50:31.244756 26697 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf025023d150485ea748b1abb4607c3b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 40325 } }
I20260806 02:50:31.244904 26697 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:31.244990 26697 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:31.245098 26697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:31.246268 26697 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf025023d150485ea748b1abb4607c3b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 40325 } }
I20260806 02:50:31.246616 26697 leader_election.cc:302] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf025023d150485ea748b1abb4607c3b; no voters:
I20260806 02:50:31.247051 26697 leader_election.cc:288] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:31.247179 26703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:31.247426 26703 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [term 1 LEADER]: Becoming Leader. State: Replica: cf025023d150485ea748b1abb4607c3b, State: Running, Role: LEADER
I20260806 02:50:31.247813 26703 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf025023d150485ea748b1abb4607c3b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 40325 } }
I20260806 02:50:31.248344 26697 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:31.249519 26704 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf025023d150485ea748b1abb4607c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf025023d150485ea748b1abb4607c3b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 40325 } } }
I20260806 02:50:31.249655 26704 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:31.250046 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf025023d150485ea748b1abb4607c3b. Latest consensus state: current_term: 1 leader_uuid: "cf025023d150485ea748b1abb4607c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf025023d150485ea748b1abb4607c3b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 40325 } } }
I20260806 02:50:31.250176 26715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:31.250197 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:31.253531 26715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:31.259514 26715 catalog_manager.cc:1347] Generated new cluster ID: 91ab2b740b45444ea8aa148455547a1d
I20260806 02:50:31.259649 26715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:31.286856 26715 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:31.288285 26715 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:31.293232 26715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: Generated new TSK 0
I20260806 02:50:31.294061 26715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:31.341569 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:40325
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:31.484766 26734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:31.485100 26734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:31.485150 26734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:31.489259 26734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:31.489341 26734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:31.489380 26734 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:50:31.489414 26734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:31.495033 26734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:40325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:31.496229 26734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:31.497277 26734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:31.507483 26744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.507478 26746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.507483 26740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.507575 26734 server_base.cc:1034] running on GCE node
I20260806 02:50:31.508327 26734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:31.509053 26734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:31.510226 26734 hybrid_clock.cc:648] HybridClock initialized: now 1785984631510213 us; error 43 us; skew 500 ppm
Aug 06 02:50:31 dist-test-slave-3kk6 krb5kdc[26569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984631, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:31.513429 26734 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:31.514410 26734 webserver.cc:460] Webserver started at http://127.24.130.129:36941/ using document root <none> and password file <none>
I20260806 02:50:31.515036 26734 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:31.515105 26734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:31.515295 26734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:31.517495 26734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "9afa0d887c17a31b0209fbf66e6dbcfc"
server_key_iv: "a273e9e45888cd8e0ec35c2cd1db63c9"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.518064 26734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "9afa0d887c17a31b0209fbf66e6dbcfc"
server_key_iv: "a273e9e45888cd8e0ec35c2cd1db63c9"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.522221 26734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:31.524976 26754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.526387 26734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:31.526712 26734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "9afa0d887c17a31b0209fbf66e6dbcfc"
server_key_iv: "a273e9e45888cd8e0ec35c2cd1db63c9"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.526862 26734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:31.550869 26734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:31.554469 26734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:31.554749 26734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:31.556419 26734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:31.556479 26734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.556525 26734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:31.556584 26734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:50:31 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984631, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:31.568892 26734 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:41227
I20260806 02:50:31.568965 26938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:41227 every 8 connection(s)
I20260806 02:50:31.570245 26734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:31.571287 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 26734
I20260806 02:50:31.571414 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:31.571691 25098 external_mini_cluster.cc:1428] Setting key b0d027a2563d89312823d1dc444796d6
I20260806 02:50:31.573962 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.557494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:42551 (local address 127.24.130.190:40325)
0806 02:50:31.557634 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.557638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.559107 (+ 1469us) server_negotiation.cc:407] Connection header received
0806 02:50:31.560399 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.560402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.560472 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.560581 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.562396 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.563436 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.565485 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.565779 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.565935 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.569492 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.569497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.569499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:31.569539 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.571471 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.572037 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.572041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.572043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.572162 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.572727 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.572730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.572739 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.572972 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.573094 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.573636 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.573794 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:50:31.574898 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.558173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.24.130.129:42551)
0806 02:50:31.558898 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:31.558975 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:31.560152 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:31.560805 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:31.560815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:31.561295 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:31.562176 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.562204 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.563636 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.563640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.565321 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.565339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.565990 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.565994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.566196 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.567096 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:31.567123 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:31.569250 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:31.571597 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.571604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.571619 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.571917 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.572464 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.572466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.572468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.572621 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.573095 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:31.573104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:31.573375 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 02:50:31.573592 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 02:50:31.573949 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":270,"threads_started":1}
I20260806 02:50:31.576874 26939 heartbeater.cc:347] Connected to a master server at 127.24.130.190:40325
I20260806 02:50:31.577081 26939 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:31.577764 26939 heartbeater.cc:511] Master 127.24.130.190:40325 requested a full tablet report, sending...
I20260806 02:50:31.579726 26632 ts_manager.cc:194] Registered new tserver with Master: 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227)
I20260806 02:50:31.580863 26632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:42551
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:31.590425 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.579538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:50623 (local address 127.24.130.190:40325)
0806 02:50:31.579662 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.579666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.582072 (+ 2406us) server_negotiation.cc:407] Connection header received
0806 02:50:31.582202 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.582205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.582255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.582348 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.583040 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.583681 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.584929 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.585150 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.585227 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.586739 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.586743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.586745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:31.586771 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.588343 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.588905 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.588908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.588909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.588990 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.589344 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.589347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.589348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.589573 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.589699 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.590054 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.590215 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:50:31.590447 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.579451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.24.130.129:50623)
0806 02:50:31.581958 (+ 2507us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:31.581977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:31.582099 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:31.582393 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:31.582396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:31.582709 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:31.582879 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.582891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.583826 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.583831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.584803 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.584817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.585284 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.585288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.585415 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.586025 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:31.586041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:31.586537 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:31.588476 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.588481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.588484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.588771 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.589092 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.589095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.589097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.589191 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.589686 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:31.589689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:31.589803 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 02:50:31.589922 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:50:31.590117 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2400,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:31.608011 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:40325
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:31.734704 26952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:31.735035 26952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:31.735085 26952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:31.739866 26952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:31.739966 26952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:31.740010 26952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:50:31.740046 26952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:31.746129 26952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:40325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:31.747375 26952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:31.748715 26952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:31.760159 26958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.760144 26959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.760219 26962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.760839 26952 server_base.cc:1034] running on GCE node
I20260806 02:50:31.761297 26952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:31.762007 26952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:31.763242 26952 hybrid_clock.cc:648] HybridClock initialized: now 1785984631763227 us; error 38 us; skew 500 ppm
Aug 06 02:50:31 dist-test-slave-3kk6 krb5kdc[26569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984631, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:31.766419 26952 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:31.767520 26952 webserver.cc:460] Webserver started at http://127.24.130.130:35419/ using document root <none> and password file <none>
I20260806 02:50:31.768137 26952 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:31.768240 26952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:31.768496 26952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:31.770807 26952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "1b70ae47f5304b23a20e90523c27ab66"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "0cabb83e91e8db80f05091e4dc83cc66"
server_key_iv: "0b1706d65a243ab58df7b94ac1b4e647"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.771450 26952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "1b70ae47f5304b23a20e90523c27ab66"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "0cabb83e91e8db80f05091e4dc83cc66"
server_key_iv: "0b1706d65a243ab58df7b94ac1b4e647"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.775712 26952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:31.778908 26977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.780511 26952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:31.780707 26952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "1b70ae47f5304b23a20e90523c27ab66"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-3kk6"
server_key: "0cabb83e91e8db80f05091e4dc83cc66"
server_key_iv: "0b1706d65a243ab58df7b94ac1b4e647"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.780870 26952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:31.792310 26952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:31.796483 26952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:31.796680 26952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:31.798308 26952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:31.798370 26952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.798414 26952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:31.798472 26952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:50:31 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984631, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:31.810906 27140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:42041 every 8 connection(s)
I20260806 02:50:31.810954 26952 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:42041
I20260806 02:50:31.812012 26952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:31.815678 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.799588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:50655 (local address 127.24.130.190:40325)
0806 02:50:31.799750 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.799754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.800977 (+ 1223us) server_negotiation.cc:407] Connection header received
0806 02:50:31.802349 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.802354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.802424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.802520 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.804465 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.805032 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.807124 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.807497 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.807655 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.811186 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.811190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.811192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:31.811224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.813303 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.813981 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.813985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.813986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.814084 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.814436 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.814438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.814440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.814729 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.814823 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.815400 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.815545 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:50:31.816565 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.800192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.24.130.130:50655)
0806 02:50:31.800782 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:31.800834 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:31.802069 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:31.802806 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:31.802814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:31.803422 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:31.804246 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.804268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.805191 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.805194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.806887 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.806908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.807697 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.807700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.807876 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.808703 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:31.808733 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:31.810921 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:31.813436 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.813441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.813454 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.813835 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.814175 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.814178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.814179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.814328 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.814842 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:31.814848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:31.815108 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:50:31.815277 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:50:31.815573 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":153,"threads_started":1}
I20260806 02:50:31.816978 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 26952
I20260806 02:50:31.817113 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:31.817487 25098 external_mini_cluster.cc:1428] Setting key 26819214bbc2f1aada7abbcef6a9e64c
I20260806 02:50:31.818019 27141 heartbeater.cc:347] Connected to a master server at 127.24.130.190:40325
I20260806 02:50:31.818219 27141 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:31.818910 27141 heartbeater.cc:511] Master 127.24.130.190:40325 requested a full tablet report, sending...
I20260806 02:50:31.822149 26633 ts_manager.cc:194] Registered new tserver with Master: 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130:42041)
I20260806 02:50:31.823107 26633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:50655
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:50:31.830435 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.822069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:43525 (local address 127.24.130.190:40325)
0806 02:50:31.822207 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.822212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.822496 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:50:31.822643 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.822646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.822721 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.822843 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.823400 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.824086 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.825083 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.825301 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.825418 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.826914 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.826917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.826920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:31.826955 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.828532 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.829014 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.829018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.829020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.829110 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:31.829453 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:31.829457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:31.829459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:31.829717 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.829838 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.830123 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.830261 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:50:31.830440 27149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.822022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.24.130.130:43525)
0806 02:50:31.822403 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:31.822417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:31.822543 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:31.822864 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:31.822867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:31.823103 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:31.823271 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.823283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.824242 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.824245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.824835 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.824844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.825438 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.825442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.825538 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.826283 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:31.826297 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:31.826759 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:31.828667 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.828671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.828674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.828901 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.829259 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:31.829261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:31.829263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:31.829348 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:31.829827 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:31.829830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:31.829938 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:50:31.830057 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:50:31.830232 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:31.851187 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:40325
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:31.973722 27153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:31.974076 27153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:31.974154 27153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:31.978825 27153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:31.978914 27153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:31.978958 27153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:50:31.978994 27153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:31.984681 27153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:40325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:31.985913 27153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:31.987020 27153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:31.997864 27164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.997918 27160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.997956 27153 server_base.cc:1034] running on GCE node
W20260806 02:50:31.998200 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.998778 27153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:31.999539 27153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:32.000742 27153 hybrid_clock.cc:648] HybridClock initialized: now 1785984632000731 us; error 34 us; skew 500 ppm
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[26569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984632, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:32.004175 27153 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:32.005267 27153 webserver.cc:460] Webserver started at http://127.24.130.131:46799/ using document root <none> and password file <none>
I20260806 02:50:32.005925 27153 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:32.006031 27153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:32.006294 27153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:32.008723 27153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "6417ff0a8a9c46a48fa36dc611904a69"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-3kk6"
server_key: "12033db4f748d4aac239c7f920f40d87"
server_key_iv: "17c7c1a3ec2f8ed3a9dd371d3fdcf90d"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.009387 27153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "6417ff0a8a9c46a48fa36dc611904a69"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-3kk6"
server_key: "12033db4f748d4aac239c7f920f40d87"
server_key_iv: "17c7c1a3ec2f8ed3a9dd371d3fdcf90d"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.013703 27153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:32.016625 27173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.018177 27153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 02:50:32.018383 27153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "6417ff0a8a9c46a48fa36dc611904a69"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-3kk6"
server_key: "12033db4f748d4aac239c7f920f40d87"
server_key_iv: "17c7c1a3ec2f8ed3a9dd371d3fdcf90d"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.018550 27153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:32.032560 27153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:32.036439 27153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:32.036628 27153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:32.038167 27153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:32.038228 27153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.038272 27153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:32.038343 27153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984632, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:32.051050 27153 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:32947
I20260806 02:50:32.051100 27336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:32947 every 8 connection(s)
I20260806 02:50:32.052525 27153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:32.054735 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.039371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:43635 (local address 127.24.130.190:40325)
0806 02:50:32.039536 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.039546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.040796 (+ 1250us) server_negotiation.cc:407] Connection header received
0806 02:50:32.042046 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.042050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.042114 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.042226 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.044113 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.045045 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.046420 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.046717 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.046846 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.050133 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.050139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.050146 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:32.050187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.052235 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.052912 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.052917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.052919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.053031 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.053453 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.053457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.053459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.053678 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.053800 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.054342 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.054498 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:50:32.055594 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.039996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.24.130.131:43635)
0806 02:50:32.040597 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.040650 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.041778 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.042454 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.042466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.043060 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:32.043886 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.043916 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.045196 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.045199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.046271 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.046282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.046912 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.046916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.047150 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.047885 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:32.047902 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:32.049907 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:32.052393 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:32.052399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:32.052412 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:32.052775 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:32.053157 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:32.053160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:32.053163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:32.053355 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:32.053796 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:32.053804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:32.054086 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.054295 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.054645 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":163,"threads_started":1}
I20260806 02:50:32.056875 27337 heartbeater.cc:347] Connected to a master server at 127.24.130.190:40325
I20260806 02:50:32.057125 27337 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:32.057768 27337 heartbeater.cc:511] Master 127.24.130.190:40325 requested a full tablet report, sending...
I20260806 02:50:32.059146 26632 ts_manager.cc:194] Registered new tserver with Master: 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947)
I20260806 02:50:32.059928 26632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:43635
I20260806 02:50:32.059973 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 27153
I20260806 02:50:32.060119 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:32.060454 25098 external_mini_cluster.cc:1428] Setting key 3829179edd62fe80e813edd30ade27ad
I20260806 02:50:32.062800 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:32.069685 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.060814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.24.130.131:47433)
0806 02:50:32.061143 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.061160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.061267 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.061561 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.061564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.062014 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:32.062127 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.062134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.063161 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.063164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.063759 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.063769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.064384 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.064388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.064487 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.065116 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:32.065154 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:32.065716 (+ 562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:32.067772 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:32.067779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:32.067784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:32.068119 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:32.068419 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:32.068425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:32.068427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:32.068546 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:32.068996 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:32.069000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:32.069126 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.069258 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.069467 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":146,"threads_started":1}
I20260806 02:50:32.069792 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.060320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:47433 (local address 127.24.130.190:40325)
0806 02:50:32.060450 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.060455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.061222 (+ 767us) server_negotiation.cc:407] Connection header received
0806 02:50:32.061380 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.061384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.061445 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.061538 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.062253 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.063035 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.063916 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.064252 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.064382 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.065903 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.065906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.065908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:32.065936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.067639 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.068233 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.068235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.068237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.068317 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.068658 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.068661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.068662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.068859 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.068969 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.069412 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.069565 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[26569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:32.095726 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.085734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.24.130.190:40325)
0806 02:50:32.085870 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.085875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.085897 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:50:32.085987 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.085990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.086045 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.086141 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.087018 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.087583 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.088504 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.088755 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.088878 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.091977 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.091982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.091984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:32.092019 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.093519 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.094049 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.094053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.094057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.094145 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.094448 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.094452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.094454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.094754 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.094906 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.095269 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.095467 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:50:32.100463 26632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:32.104776 26632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:32.121277 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.114712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32947 (local address 127.0.0.1:48414)
0806 02:50:32.115639 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.115703 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.115864 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.116525 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.116532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.116557 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.116720 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.116759 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.118082 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.118086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.119551 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.119561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.120624 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.120628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.120748 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.120796 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.120933 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.121095 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":56,"thread_start_us":97,"threads_started":1}
I20260806 02:50:32.121295 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.114857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.24.130.131:32947)
0806 02:50:32.115535 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.115549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.116021 (+ 472us) server_negotiation.cc:407] Connection header received
0806 02:50:32.116110 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.116115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.116357 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.116534 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.116896 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.117962 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.119687 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.120471 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.120630 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.120787 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.120987 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.121101 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":130,"threads_started":1}
I20260806 02:50:32.121721 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.115542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42041 (local address 127.0.0.1:33934)
0806 02:50:32.116657 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.116670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.116778 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.117232 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.117235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.117255 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.117367 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.117374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.118814 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.118817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.120017 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.120028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.121267 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.121270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.121381 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.121413 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.121495 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.121573 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":20,"thread_start_us":101,"threads_started":1}
I20260806 02:50:32.122129 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.114871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:41227 (local address 127.0.0.1:38586)
0806 02:50:32.115891 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.115909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.116090 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.116815 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.116819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.116839 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.116958 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.116963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.118486 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.118527 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.119918 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.119931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.121222 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.121226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.121342 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.121813 (+ 471us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.121914 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.121989 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":91,"thread_start_us":110,"threads_started":1}
I20260806 02:50:32.122241 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.116055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.24.130.130:42041)
0806 02:50:32.116507 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.116518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.116785 (+ 267us) server_negotiation.cc:407] Connection header received
0806 02:50:32.116890 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.116895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.117084 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.117240 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.117519 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.118641 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.120124 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.120917 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.121068 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.121198 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.121994 (+ 796us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.122102 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":127,"threads_started":1}
I20260806 02:50:32.122628 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.115049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.24.130.129:41227)
0806 02:50:32.115735 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.115744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.116007 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:50:32.116262 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.116268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.116482 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.116619 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.117078 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.118342 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.120071 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.120875 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.121027 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.121150 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.122379 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.122490 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":103,"threads_started":1}
I20260806 02:50:32.125003 26837 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5218556a3c4367b198b055a29a08e4 (DEFAULT_TABLE table=test-table [id=19f5e2d493d14a9f95a850c1e10b5554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.125010 27251 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5218556a3c4367b198b055a29a08e4 (DEFAULT_TABLE table=test-table [id=19f5e2d493d14a9f95a850c1e10b5554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.125005 27044 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5218556a3c4367b198b055a29a08e4 (DEFAULT_TABLE table=test-table [id=19f5e2d493d14a9f95a850c1e10b5554]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.126120 27251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5218556a3c4367b198b055a29a08e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.126116 27044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5218556a3c4367b198b055a29a08e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.126096 26837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5218556a3c4367b198b055a29a08e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.133261 27365 tablet_bootstrap.cc:492] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Bootstrap starting.
I20260806 02:50:32.135064 27366 tablet_bootstrap.cc:492] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Bootstrap starting.
I20260806 02:50:32.135830 27365 tablet_bootstrap.cc:654] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.136648 27365 log.cc:824] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:32.137020 27366 tablet_bootstrap.cc:654] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.137681 27368 tablet_bootstrap.cc:492] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Bootstrap starting.
I20260806 02:50:32.137918 27366 log.cc:824] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:32.139067 27365 tablet_bootstrap.cc:492] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: No bootstrap required, opened a new log
I20260806 02:50:32.139297 27365 ts_tablet_manager.cc:1397] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:50:32.139941 27368 tablet_bootstrap.cc:654] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.140540 27366 tablet_bootstrap.cc:492] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: No bootstrap required, opened a new log
I20260806 02:50:32.140796 27366 ts_tablet_manager.cc:1397] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260806 02:50:32.140812 27368 log.cc:824] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:32.143143 27368 tablet_bootstrap.cc:492] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: No bootstrap required, opened a new log
I20260806 02:50:32.143366 27368 ts_tablet_manager.cc:1397] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:50:32.143501 27365 raft_consensus.cc:348] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.143781 27365 raft_consensus.cc:374] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.143836 27365 raft_consensus.cc:729] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1efbcfb69e49bb9c26c2a217fec7ea, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.144352 27365 consensus_queue.cc:260] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.144843 27366 raft_consensus.cc:348] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.145102 27366 raft_consensus.cc:374] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.145149 27366 raft_consensus.cc:729] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6417ff0a8a9c46a48fa36dc611904a69, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.145496 27365 ts_tablet_manager.cc:1428] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:32.145617 27366 consensus_queue.cc:260] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.145892 26939 heartbeater.cc:503] Master 127.24.130.190:40325 was elected leader, sending a full tablet report...
I20260806 02:50:32.146380 27337 heartbeater.cc:503] Master 127.24.130.190:40325 was elected leader, sending a full tablet report...
I20260806 02:50:32.146689 27366 ts_tablet_manager.cc:1428] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:50:32.147430 27368 raft_consensus.cc:348] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.147702 27368 raft_consensus.cc:374] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.147766 27368 raft_consensus.cc:729] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b70ae47f5304b23a20e90523c27ab66, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.148195 27368 consensus_queue.cc:260] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.148960 27141 heartbeater.cc:503] Master 127.24.130.190:40325 was elected leader, sending a full tablet report...
I20260806 02:50:32.149359 27368 ts_tablet_manager.cc:1428] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 02:50:32.246439 27377 raft_consensus.cc:482] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:32.246735 27377 raft_consensus.cc:504] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.247980 27377 leader_election.cc:288] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947), 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227)
I20260806 02:50:32.252518 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.248515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:57785 (local address 127.24.130.129:41227)
0806 02:50:32.248739 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.248750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.248887 (+ 137us) server_negotiation.cc:407] Connection header received
0806 02:50:32.249039 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.249041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.249089 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.249184 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.249493 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.250153 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.251289 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.251957 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.252077 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.252125 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.252344 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.252408 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:50:32.252554 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.248364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:41227 (local address 127.24.130.130:57785)
0806 02:50:32.248781 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.248797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.248923 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.249206 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.249208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.249224 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.249366 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.249374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.250275 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.250278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.251143 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.251154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.252084 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.252088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.252206 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.252236 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.252318 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.252390 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":146,"threads_started":1}
I20260806 02:50:32.253213 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.248529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:36129 (local address 127.24.130.131:32947)
0806 02:50:32.248727 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.248733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.248783 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:50:32.248849 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.248852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.248908 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.248990 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.249384 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.250011 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.252002 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.252612 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.252742 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.252792 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.253022 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.253115 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 02:50:32.253221 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.248269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32947 (local address 127.24.130.130:36129)
0806 02:50:32.248401 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.248420 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.248533 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.249038 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.249041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.249068 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.249232 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.249241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.250138 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.250142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.251854 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.251866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.252749 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.252753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.252883 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.252904 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.252977 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.253083 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 02:50:32.253435 26870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5218556a3c4367b198b055a29a08e4" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" is_pre_election: true
I20260806 02:50:32.253810 26870 raft_consensus.cc:2454] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 0.
I20260806 02:50:32.253811 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5218556a3c4367b198b055a29a08e4" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6417ff0a8a9c46a48fa36dc611904a69" is_pre_election: true
I20260806 02:50:32.254061 27278 raft_consensus.cc:2454] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 0.
I20260806 02:50:32.254366 26978 leader_election.cc:302] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b70ae47f5304b23a20e90523c27ab66, 7d1efbcfb69e49bb9c26c2a217fec7ea; no voters:
I20260806 02:50:32.254738 27377 raft_consensus.cc:2781] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:32.254848 27377 raft_consensus.cc:482] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:32.254911 27377 raft_consensus.cc:3037] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.256075 27377 raft_consensus.cc:504] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.256531 27377 leader_election.cc:288] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 election: Requested vote from peers 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947), 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227)
I20260806 02:50:32.256968 26870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5218556a3c4367b198b055a29a08e4" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea"
I20260806 02:50:32.257030 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5218556a3c4367b198b055a29a08e4" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6417ff0a8a9c46a48fa36dc611904a69"
I20260806 02:50:32.257141 26870 raft_consensus.cc:3037] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.257220 27278 raft_consensus.cc:3037] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.258481 26870 raft_consensus.cc:2454] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 1.
I20260806 02:50:32.258925 26978 leader_election.cc:302] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b70ae47f5304b23a20e90523c27ab66, 7d1efbcfb69e49bb9c26c2a217fec7ea; no voters:
I20260806 02:50:32.259137 27377 raft_consensus.cc:2781] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:32.259243 27278 raft_consensus.cc:2454] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 1.
I20260806 02:50:32.259438 27377 raft_consensus.cc:686] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 LEADER]: Becoming Leader. State: Replica: 1b70ae47f5304b23a20e90523c27ab66, State: Running, Role: LEADER
I20260806 02:50:32.259850 27377 consensus_queue.cc:237] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } }
I20260806 02:50:32.263762 26631 catalog_manager.cc:5495] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130). New cstate: current_term: 1 leader_uuid: "1b70ae47f5304b23a20e90523c27ab66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } health_report { overall_health: HEALTHY } } }
I20260806 02:50:32.282585 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.278391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.24.130.130:42041)
0806 02:50:32.278556 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.278560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.278576 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:50:32.278818 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.278821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.278885 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.278976 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:32.279344 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.280083 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.281024 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.281246 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.281327 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.281651 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.282131 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.282222 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.282319 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.282379 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:50:32.289327 26870 raft_consensus.cc:1261] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Refusing update from remote peer 1b70ae47f5304b23a20e90523c27ab66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:32.289327 27278 raft_consensus.cc:1261] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Refusing update from remote peer 1b70ae47f5304b23a20e90523c27ab66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:32.290150 27377 consensus_queue.cc:1035] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:32.290397 27379 consensus_queue.cc:1035] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:32.298305 27384 mvcc.cc:205] Tried to move back new op lower bound from 7315393053849042944 to 7315393053739585536. Current Snapshot: MvccSnapshot[applied={T|T < 7315393053849042944}]
I20260806 02:50:32.301124 27388 mvcc.cc:205] Tried to move back new op lower bound from 7315393053849042944 to 7315393053739585536. Current Snapshot: MvccSnapshot[applied={T|T < 7315393053849042944}]
I20260806 02:50:32.301471 27387 mvcc.cc:205] Tried to move back new op lower bound from 7315393053849042944 to 7315393053739585536. Current Snapshot: MvccSnapshot[applied={T|T < 7315393053849042944 or (T in {7315393053849042944})}]
I20260806 02:50:32.311587 26632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:32.311830 26632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:32.314424 26632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: Sending DeleteTablet for 3 replicas of tablet 5e5218556a3c4367b198b055a29a08e4
I20260806 02:50:32.315414 27251 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5218556a3c4367b198b055a29a08e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48414
I20260806 02:50:32.315716 27044 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5218556a3c4367b198b055a29a08e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:33934
I20260806 02:50:32.316063 26837 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5218556a3c4367b198b055a29a08e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:38586
I20260806 02:50:32.316531 27398 tablet_replica.cc:330] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: stopping tablet replica
I20260806 02:50:32.316583 27400 tablet_replica.cc:330] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: stopping tablet replica
I20260806 02:50:32.316567 26632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:50:32.316960 27400 raft_consensus.cc:2229] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 02:50:32.317090 26632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:32.317140 27398 raft_consensus.cc:2229] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:32.317353 27400 raft_consensus.cc:2258] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:32.317660 27398 raft_consensus.cc:2258] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:32.318186 27399 tablet_replica.cc:330] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: stopping tablet replica
I20260806 02:50:32.318506 27399 raft_consensus.cc:2229] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:32.319002 27399 raft_consensus.cc:2258] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:32.319967 27398 ts_tablet_manager.cc:1905] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:32.320129 27400 ts_tablet_manager.cc:1905] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:32.323140 27398 ts_tablet_manager.cc:1918] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:32.323179 27400 ts_tablet_manager.cc:1918] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:32.323279 27400 log.cc:1196] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/5e5218556a3c4367b198b055a29a08e4
I20260806 02:50:32.323526 27398 log.cc:1196] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/5e5218556a3c4367b198b055a29a08e4
I20260806 02:50:32.323635 27400 ts_tablet_manager.cc:1939] T 5e5218556a3c4367b198b055a29a08e4 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Deleting consensus metadata
I20260806 02:50:32.323980 27399 ts_tablet_manager.cc:1905] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:32.324137 27398 ts_tablet_manager.cc:1939] T 5e5218556a3c4367b198b055a29a08e4 P 6417ff0a8a9c46a48fa36dc611904a69: Deleting consensus metadata
I20260806 02:50:32.325048 26611 catalog_manager.cc:4841] TS 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227): tablet 5e5218556a3c4367b198b055a29a08e4 (table test-table [id=19f5e2d493d14a9f95a850c1e10b5554]) successfully deleted
I20260806 02:50:32.325387 26618 catalog_manager.cc:4841] TS 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947): tablet 5e5218556a3c4367b198b055a29a08e4 (table test-table [id=19f5e2d493d14a9f95a850c1e10b5554]) successfully deleted
I20260806 02:50:32.325732 27251 tablet_service.cc:1467] Processing CreateTablet for tablet 31972b8e06fd40ee8bc688bd4ad35764 (DEFAULT_TABLE table=test-table [id=05e52add77b041e785b322df46ad6937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.325826 27044 tablet_service.cc:1467] Processing CreateTablet for tablet 31972b8e06fd40ee8bc688bd4ad35764 (DEFAULT_TABLE table=test-table [id=05e52add77b041e785b322df46ad6937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.325959 27251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31972b8e06fd40ee8bc688bd4ad35764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.326088 27044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31972b8e06fd40ee8bc688bd4ad35764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.326941 27399 ts_tablet_manager.cc:1918] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:32.327039 27399 log.cc:1196] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/5e5218556a3c4367b198b055a29a08e4
I20260806 02:50:32.327399 27399 ts_tablet_manager.cc:1939] T 5e5218556a3c4367b198b055a29a08e4 P 1b70ae47f5304b23a20e90523c27ab66: Deleting consensus metadata
I20260806 02:50:32.328197 27366 tablet_bootstrap.cc:492] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Bootstrap starting.
I20260806 02:50:32.328495 27368 tablet_bootstrap.cc:492] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Bootstrap starting.
I20260806 02:50:32.328894 26611 catalog_manager.cc:4841] TS 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130:42041): tablet 5e5218556a3c4367b198b055a29a08e4 (table test-table [id=19f5e2d493d14a9f95a850c1e10b5554]) successfully deleted
I20260806 02:50:32.329268 27366 tablet_bootstrap.cc:654] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.329347 26837 tablet_service.cc:1467] Processing CreateTablet for tablet 31972b8e06fd40ee8bc688bd4ad35764 (DEFAULT_TABLE table=test-table [id=05e52add77b041e785b322df46ad6937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.329511 27368 tablet_bootstrap.cc:654] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.329617 26837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31972b8e06fd40ee8bc688bd4ad35764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.331539 27368 tablet_bootstrap.cc:492] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: No bootstrap required, opened a new log
I20260806 02:50:32.331539 27366 tablet_bootstrap.cc:492] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: No bootstrap required, opened a new log
I20260806 02:50:32.331636 27366 ts_tablet_manager.cc:1397] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:32.331636 27368 ts_tablet_manager.cc:1397] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 02:50:32.332157 27366 raft_consensus.cc:348] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.332157 27368 raft_consensus.cc:348] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.332261 27365 tablet_bootstrap.cc:492] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Bootstrap starting.
I20260806 02:50:32.332280 27366 raft_consensus.cc:374] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.332280 27368 raft_consensus.cc:374] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.332316 27366 raft_consensus.cc:729] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6417ff0a8a9c46a48fa36dc611904a69, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.332316 27368 raft_consensus.cc:729] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b70ae47f5304b23a20e90523c27ab66, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.332453 27366 consensus_queue.cc:260] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.332455 27368 consensus_queue.cc:260] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.332716 27366 ts_tablet_manager.cc:1428] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:32.332734 27368 ts_tablet_manager.cc:1428] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:32.333324 27365 tablet_bootstrap.cc:654] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.335584 27365 tablet_bootstrap.cc:492] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: No bootstrap required, opened a new log
I20260806 02:50:32.335726 27365 ts_tablet_manager.cc:1397] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:50:32.336436 27365 raft_consensus.cc:348] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.336581 27365 raft_consensus.cc:374] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.336629 27365 raft_consensus.cc:729] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1efbcfb69e49bb9c26c2a217fec7ea, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.336777 27365 consensus_queue.cc:260] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.337080 27365 ts_tablet_manager.cc:1428] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:50:32.440600 27394 raft_consensus.cc:482] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:32.440778 27394 raft_consensus.cc:504] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.441347 27394 leader_election.cc:288] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227), 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947)
I20260806 02:50:32.442005 26869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31972b8e06fd40ee8bc688bd4ad35764" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" is_pre_election: true
I20260806 02:50:32.442020 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31972b8e06fd40ee8bc688bd4ad35764" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6417ff0a8a9c46a48fa36dc611904a69" is_pre_election: true
I20260806 02:50:32.442206 27278 raft_consensus.cc:2454] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 0.
I20260806 02:50:32.442198 26869 raft_consensus.cc:2454] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 0.
I20260806 02:50:32.442626 26978 leader_election.cc:302] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b70ae47f5304b23a20e90523c27ab66, 7d1efbcfb69e49bb9c26c2a217fec7ea; no voters:
I20260806 02:50:32.442867 27394 raft_consensus.cc:2781] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:32.442946 27394 raft_consensus.cc:482] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:32.443005 27394 raft_consensus.cc:3037] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.444137 27394 raft_consensus.cc:504] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.444569 27394 leader_election.cc:288] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227), 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947)
I20260806 02:50:32.445063 27278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31972b8e06fd40ee8bc688bd4ad35764" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6417ff0a8a9c46a48fa36dc611904a69"
I20260806 02:50:32.445050 26869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31972b8e06fd40ee8bc688bd4ad35764" candidate_uuid: "1b70ae47f5304b23a20e90523c27ab66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea"
I20260806 02:50:32.445214 27278 raft_consensus.cc:3037] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.445228 26869 raft_consensus.cc:3037] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.446235 27278 raft_consensus.cc:2454] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 1.
I20260806 02:50:32.446609 26869 raft_consensus.cc:2454] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b70ae47f5304b23a20e90523c27ab66 in term 1.
I20260806 02:50:32.446633 26985 leader_election.cc:302] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b70ae47f5304b23a20e90523c27ab66, 6417ff0a8a9c46a48fa36dc611904a69; no voters:
I20260806 02:50:32.446906 27394 raft_consensus.cc:2781] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:32.447003 27394 raft_consensus.cc:686] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 LEADER]: Becoming Leader. State: Replica: 1b70ae47f5304b23a20e90523c27ab66, State: Running, Role: LEADER
I20260806 02:50:32.447153 27394 consensus_queue.cc:237] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.449187 26633 catalog_manager.cc:5495] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130). New cstate: current_term: 1 leader_uuid: "1b70ae47f5304b23a20e90523c27ab66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984631, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:32.510046 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.500804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:40325 (local address 127.0.0.1:57902)
0806 02:50:32.501203 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.501227 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.501350 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.501618 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.501620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.502054 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:32.502159 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.502165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.502989 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.502992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.503562 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.503570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.504027 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.504030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.504107 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.504798 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:32.504810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:32.506477 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:32.508459 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:32.508470 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:32.508476 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:32.508742 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:32.509002 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:32.509005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:32.509006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:32.509088 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:32.509476 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:32.509482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:32.509601 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.509707 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.509851 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":175,"threads_started":1}
I20260806 02:50:32.510049 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.500897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.24.130.190:40325)
0806 02:50:32.501049 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.501053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.501323 (+ 270us) server_negotiation.cc:407] Connection header received
0806 02:50:32.501438 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.501440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.501484 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.501571 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.502310 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.502875 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.503683 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.503902 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.503989 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.506642 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.506698 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.506701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:32.506733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.508330 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.508836 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.508839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.508840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.508917 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.509169 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.509171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.509173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.509374 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.509452 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.509766 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.509917 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":61}
I20260806 02:50:32.515013 26633 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:57902:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:50:32.515825 26633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:32.521224 26837 tablet_service.cc:1467] Processing CreateTablet for tablet a401634a6247440dbdaa4784b22e907f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d8d91a3d444c7e945342eade6dbee0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:50:32.521214 27044 tablet_service.cc:1467] Processing CreateTablet for tablet a401634a6247440dbdaa4784b22e907f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d8d91a3d444c7e945342eade6dbee0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:50:32.521292 27251 tablet_service.cc:1467] Processing CreateTablet for tablet a401634a6247440dbdaa4784b22e907f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d8d91a3d444c7e945342eade6dbee0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:50:32.521672 26837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a401634a6247440dbdaa4784b22e907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.521672 27044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a401634a6247440dbdaa4784b22e907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.521766 27251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a401634a6247440dbdaa4784b22e907f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.524818 27366 tablet_bootstrap.cc:492] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: Bootstrap starting.
I20260806 02:50:32.524869 27368 tablet_bootstrap.cc:492] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: Bootstrap starting.
I20260806 02:50:32.524887 27365 tablet_bootstrap.cc:492] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: Bootstrap starting.
I20260806 02:50:32.525869 27366 tablet_bootstrap.cc:654] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.525871 27368 tablet_bootstrap.cc:654] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.525872 27365 tablet_bootstrap.cc:654] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.527484 27366 tablet_bootstrap.cc:492] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: No bootstrap required, opened a new log
I20260806 02:50:32.527484 27368 tablet_bootstrap.cc:492] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: No bootstrap required, opened a new log
I20260806 02:50:32.527585 27366 ts_tablet_manager.cc:1397] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:32.527585 27368 ts_tablet_manager.cc:1397] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 02:50:32.527863 27365 tablet_bootstrap.cc:492] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: No bootstrap required, opened a new log
I20260806 02:50:32.527974 27365 ts_tablet_manager.cc:1397] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:32.528039 27366 raft_consensus.cc:348] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.528039 27368 raft_consensus.cc:348] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.528175 27366 raft_consensus.cc:374] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.528175 27368 raft_consensus.cc:374] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.528218 27366 raft_consensus.cc:729] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6417ff0a8a9c46a48fa36dc611904a69, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.528228 27368 raft_consensus.cc:729] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b70ae47f5304b23a20e90523c27ab66, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.528352 27366 consensus_queue.cc:260] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.528352 27368 consensus_queue.cc:260] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.528501 27365 raft_consensus.cc:348] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.528620 27365 raft_consensus.cc:374] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.528652 27365 raft_consensus.cc:729] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1efbcfb69e49bb9c26c2a217fec7ea, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.528666 27368 ts_tablet_manager.cc:1428] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:32.528770 27365 consensus_queue.cc:260] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.528676 27394 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.528911 27394 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.529067 27365 ts_tablet_manager.cc:1428] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:32.529048 27373 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.529202 27373 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.529428 27374 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.529597 27374 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.529865 27366 ts_tablet_manager.cc:1428] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260806 02:50:32.554533 27338 tablet.cc:2369] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:32.564193 27142 tablet.cc:2369] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:32.572199 26942 tablet.cc:2369] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:32.626557 27374 raft_consensus.cc:482] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:32.626770 27374 raft_consensus.cc:504] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.628016 27374 leader_election.cc:288] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227), 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130:42041)
I20260806 02:50:32.633154 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.628536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:55411 (local address 127.24.130.129:41227)
0806 02:50:32.628721 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.628726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.628893 (+ 167us) server_negotiation.cc:407] Connection header received
0806 02:50:32.629171 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.629176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.629235 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.629315 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.629770 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.630418 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.632056 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.632501 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.632650 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.632711 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.632928 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.633013 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:50:32.633170 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.628390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:41227 (local address 127.24.130.131:55411)
0806 02:50:32.628790 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.628810 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.628996 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.629491 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.629496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.629521 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.629631 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.629636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.630573 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.630577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.631919 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.631931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.632650 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.632654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.632766 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.632801 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.632905 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.632995 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":146,"threads_started":1}
I20260806 02:50:32.633286 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.629069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:59255 (local address 127.24.130.130:42041)
0806 02:50:32.629206 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.629211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.629684 (+ 473us) server_negotiation.cc:407] Connection header received
0806 02:50:32.629755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.629759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.629817 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.629906 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.630205 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.630876 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.632242 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.632658 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.632799 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.632853 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.633065 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.633145 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:50:32.633536 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.628927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42041 (local address 127.24.130.131:59255)
0806 02:50:32.629272 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.629289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.629402 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.629924 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.629927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.629946 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.630066 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.630071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.631020 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.631023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.632113 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.632121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.632793 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.632797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.632929 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.632950 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.633044 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.633120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":92,"threads_started":1}
I20260806 02:50:32.633984 26869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a401634a6247440dbdaa4784b22e907f" candidate_uuid: "6417ff0a8a9c46a48fa36dc611904a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" is_pre_election: true
I20260806 02:50:32.634177 26869 raft_consensus.cc:2454] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6417ff0a8a9c46a48fa36dc611904a69 in term 0.
I20260806 02:50:32.634121 27079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a401634a6247440dbdaa4784b22e907f" candidate_uuid: "6417ff0a8a9c46a48fa36dc611904a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b70ae47f5304b23a20e90523c27ab66" is_pre_election: true
I20260806 02:50:32.634457 27079 raft_consensus.cc:2454] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6417ff0a8a9c46a48fa36dc611904a69 in term 0.
I20260806 02:50:32.634608 27174 leader_election.cc:302] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6417ff0a8a9c46a48fa36dc611904a69, 7d1efbcfb69e49bb9c26c2a217fec7ea; no voters:
I20260806 02:50:32.635021 27374 raft_consensus.cc:2781] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:32.635113 27374 raft_consensus.cc:482] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:32.635174 27374 raft_consensus.cc:3037] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.636983 27374 raft_consensus.cc:504] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.637418 27374 leader_election.cc:288] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227), 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130:42041)
I20260806 02:50:32.637961 27079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a401634a6247440dbdaa4784b22e907f" candidate_uuid: "6417ff0a8a9c46a48fa36dc611904a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b70ae47f5304b23a20e90523c27ab66"
I20260806 02:50:32.637961 26869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a401634a6247440dbdaa4784b22e907f" candidate_uuid: "6417ff0a8a9c46a48fa36dc611904a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea"
I20260806 02:50:32.638118 27079 raft_consensus.cc:3037] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.638118 26869 raft_consensus.cc:3037] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:32.639420 27079 raft_consensus.cc:2454] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6417ff0a8a9c46a48fa36dc611904a69 in term 1.
I20260806 02:50:32.639420 26869 raft_consensus.cc:2454] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6417ff0a8a9c46a48fa36dc611904a69 in term 1.
I20260806 02:50:32.639762 27174 leader_election.cc:302] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b70ae47f5304b23a20e90523c27ab66, 6417ff0a8a9c46a48fa36dc611904a69; no voters:
I20260806 02:50:32.640061 27374 raft_consensus.cc:2781] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:32.640291 27374 raft_consensus.cc:686] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 LEADER]: Becoming Leader. State: Replica: 6417ff0a8a9c46a48fa36dc611904a69, State: Running, Role: LEADER
I20260806 02:50:32.640592 27374 consensus_queue.cc:237] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } }
I20260806 02:50:32.641907 27424 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: TxnStatusTablet state changed. Reason: New leader 6417ff0a8a9c46a48fa36dc611904a69. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.642036 27424 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:50:32.642395 27427 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:50:32.642599 27427 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:50:32.643527 26633 catalog_manager.cc:5495] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } health_report { overall_health: HEALTHY } } }
I20260806 02:50:32.666977 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.661946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.24.130.131:32947)
0806 02:50:32.662088 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.662092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.662109 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:50:32.662394 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.662397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.662475 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.662581 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:32.662967 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.663510 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.664877 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.665114 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.665234 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.665644 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.666307 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.666427 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.666747 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.666856 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:50:32.666962 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.661832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32947 (local address 127.0.0.1:48426)
0806 02:50:32.662005 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.662024 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.662158 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.662633 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.662637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.662646 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:32.662835 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.662842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.663648 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.663652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.664687 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.664701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.665270 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.665273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.665376 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.665453 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.666468 (+ 1015us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.666499 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.666603 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.666766 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:50:32.670586 26869 raft_consensus.cc:1261] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Refusing update from remote peer 6417ff0a8a9c46a48fa36dc611904a69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:32.670987 27079 raft_consensus.cc:1261] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Refusing update from remote peer 6417ff0a8a9c46a48fa36dc611904a69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:32.671270 27374 consensus_queue.cc:1035] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:32.671523 27424 consensus_queue.cc:1035] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:32.673209 27373 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: TxnStatusTablet state changed. Reason: New leader 6417ff0a8a9c46a48fa36dc611904a69. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.673339 27373 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.673449 27394 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: TxnStatusTablet state changed. Reason: New leader 6417ff0a8a9c46a48fa36dc611904a69. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.673579 27394 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.676002 27424 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.676142 27424 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:50:32.676558 27394 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.676688 27394 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 1b70ae47f5304b23a20e90523c27ab66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.678428 27374 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.678568 27374 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 6417ff0a8a9c46a48fa36dc611904a69: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:50:32.679951 27373 tablet_replica.cc:439] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6417ff0a8a9c46a48fa36dc611904a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 } } peers { permanent_uuid: "1b70ae47f5304b23a20e90523c27ab66" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42041 } } peers { permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 } } }
I20260806 02:50:32.680091 27373 tablet_replica.cc:442] T a401634a6247440dbdaa4784b22e907f P 7d1efbcfb69e49bb9c26c2a217fec7ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:32.696730 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.692965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32947 (local address 127.24.130.130:57199)
0806 02:50:32.693389 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.693404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.693553 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.693837 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.693840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.693847 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:32.693971 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.693977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.694779 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.694783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.695430 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.695439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.695914 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.695918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.696005 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.696055 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.696386 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.696396 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.696479 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.696575 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":120,"threads_started":1}
I20260806 02:50:32.696717 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.693079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:57199 (local address 127.24.130.131:32947)
0806 02:50:32.693215 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.693218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.693499 (+ 281us) server_negotiation.cc:407] Connection header received
0806 02:50:32.693675 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.693679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.693736 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.693824 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:32.694092 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.694620 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.695566 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.695790 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.695895 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.696213 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.696293 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.696378 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.696504 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.696560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:50:32.705153 27278 raft_consensus.cc:1261] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Refusing update from remote peer 1b70ae47f5304b23a20e90523c27ab66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:32.705134 26870 raft_consensus.cc:1261] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Refusing update from remote peer 1b70ae47f5304b23a20e90523c27ab66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:32.705709 27394 consensus_queue.cc:1035] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1efbcfb69e49bb9c26c2a217fec7ea" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 41227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:32.705911 27377 consensus_queue.cc:1035] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6417ff0a8a9c46a48fa36dc611904a69" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:32.734513 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.726829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42041 (local address 127.24.130.131:48781)
0806 02:50:32.730473 (+ 3644us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.730492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.730616 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.730963 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.730966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.730977 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:32.731110 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.731120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.732063 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.732066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.733007 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.733019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.733624 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.733627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.733723 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.733799 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.734213 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.734218 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.734286 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.734358 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3498,"thread_start_us":119,"threads_started":1}
I20260806 02:50:32.734504 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.727015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:48781 (local address 127.24.130.130:42041)
0806 02:50:32.727130 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.727141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.730611 (+ 3470us) server_negotiation.cc:407] Connection header received
0806 02:50:32.730777 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.730782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.730845 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.730931 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:32.731233 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.731913 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.733136 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.733500 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.733612 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.734025 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:32.734120 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:32.734199 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.734296 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.734359 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 02:50:32.761155 26633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:32.761451 26633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:32.763935 26633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf025023d150485ea748b1abb4607c3b: Sending DeleteTablet for 3 replicas of tablet 31972b8e06fd40ee8bc688bd4ad35764
I20260806 02:50:32.764684 27251 tablet_service.cc:1514] Processing DeleteTablet for tablet 31972b8e06fd40ee8bc688bd4ad35764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48414
I20260806 02:50:32.764755 26837 tablet_service.cc:1514] Processing DeleteTablet for tablet 31972b8e06fd40ee8bc688bd4ad35764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:38586
I20260806 02:50:32.764968 27400 tablet_replica.cc:330] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: stopping tablet replica
I20260806 02:50:32.764976 27398 tablet_replica.cc:330] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: stopping tablet replica
I20260806 02:50:32.765120 27400 raft_consensus.cc:2229] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:32.765113 27398 raft_consensus.cc:2229] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:32.765270 27400 raft_consensus.cc:2258] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:32.765269 27398 raft_consensus.cc:2258] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:32.765905 27044 tablet_service.cc:1514] Processing DeleteTablet for tablet 31972b8e06fd40ee8bc688bd4ad35764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:33934
I20260806 02:50:32.766117 27399 tablet_replica.cc:330] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: stopping tablet replica
I20260806 02:50:32.766255 27399 raft_consensus.cc:2229] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:32.766467 27399 raft_consensus.cc:2258] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:32.766479 27400 ts_tablet_manager.cc:1905] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:32.767215 27398 ts_tablet_manager.cc:1905] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:32.767359 27399 ts_tablet_manager.cc:1905] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:32.769829 27400 ts_tablet_manager.cc:1918] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:32.769912 27400 log.cc:1196] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/31972b8e06fd40ee8bc688bd4ad35764
I20260806 02:50:32.770161 27398 ts_tablet_manager.cc:1918] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:32.770198 27400 ts_tablet_manager.cc:1939] T 31972b8e06fd40ee8bc688bd4ad35764 P 7d1efbcfb69e49bb9c26c2a217fec7ea: Deleting consensus metadata
I20260806 02:50:32.770251 27398 log.cc:1196] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/31972b8e06fd40ee8bc688bd4ad35764
I20260806 02:50:32.770246 27399 ts_tablet_manager.cc:1918] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:32.770323 27399 log.cc:1196] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/31972b8e06fd40ee8bc688bd4ad35764
I20260806 02:50:32.770507 27398 ts_tablet_manager.cc:1939] T 31972b8e06fd40ee8bc688bd4ad35764 P 6417ff0a8a9c46a48fa36dc611904a69: Deleting consensus metadata
I20260806 02:50:32.770591 27399 ts_tablet_manager.cc:1939] T 31972b8e06fd40ee8bc688bd4ad35764 P 1b70ae47f5304b23a20e90523c27ab66: Deleting consensus metadata
I20260806 02:50:32.771183 26611 catalog_manager.cc:4841] TS 7d1efbcfb69e49bb9c26c2a217fec7ea (127.24.130.129:41227): tablet 31972b8e06fd40ee8bc688bd4ad35764 (table test-table [id=05e52add77b041e785b322df46ad6937]) successfully deleted
I20260806 02:50:32.771190 26618 catalog_manager.cc:4841] TS 6417ff0a8a9c46a48fa36dc611904a69 (127.24.130.131:32947): tablet 31972b8e06fd40ee8bc688bd4ad35764 (table test-table [id=05e52add77b041e785b322df46ad6937]) successfully deleted
I20260806 02:50:32.771428 26611 catalog_manager.cc:4841] TS 1b70ae47f5304b23a20e90523c27ab66 (127.24.130.130:42041): tablet 31972b8e06fd40ee8bc688bd4ad35764 (table test-table [id=05e52add77b041e785b322df46ad6937]) successfully deleted
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[26569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:32.787262 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.772274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.24.130.129:41227)
0806 02:50:32.772438 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.772443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.772706 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:50:32.772846 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.772848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.772894 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.772965 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.773971 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.774543 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.775915 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.776191 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.776287 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.782158 (+ 5871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.782176 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.782192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:32.782232 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.784768 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.785553 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.785561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.785563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.785681 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.786114 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.786117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.786118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.786434 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.786616 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.786968 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.787090 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:50:32.788064 26933 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38598
I20260806 02:50:32.799551 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.790158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.24.130.190:40325)
0806 02:50:32.790348 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.790354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.790703 (+ 349us) server_negotiation.cc:407] Connection header received
0806 02:50:32.790904 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.790908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.790970 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.791058 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.791787 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.792564 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.793438 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.793788 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.793906 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.795787 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.795791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.795794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:32.795828 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.797592 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.798067 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.798071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.798073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.798179 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:32.798473 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:32.798476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:32.798478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:32.798821 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.798958 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.799226 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.799373 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260806 02:50:32.800098 26633 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57910
I20260806 02:50:32.801502 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 26734
I20260806 02:50:32.814519 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 26952
I20260806 02:50:32.828660 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 27153
I20260806 02:50:32.843681 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 26590
2026-08-06T02:50:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3216 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[27456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[27456](info): set up 2 sockets
Aug 06 02:50:32 dist-test-slave-3kk6 krb5kdc[27456](info): commencing operation
krb5kdc: starting...
W20260806 02:50:34.094049 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:34.241464 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43523
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:34.361399 27473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:34.361727 27473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:34.361827 27473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:34.366180 27473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:34.366259 27473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:50:34.366277 27473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:34.366328 27473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:34.366369 27473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:34.366397 27473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:34.372051 27473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43523
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:34.373236 27473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:34.374382 27473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:34.383823 27473 server_base.cc:1034] running on GCE node
W20260806 02:50:34.383749 27483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:34.383750 27479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:34.383756 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:34.384697 27473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:34.385869 27473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:34.387059 27473 hybrid_clock.cc:648] HybridClock initialized: now 1785984634387033 us; error 43 us; skew 500 ppm
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:34.390067 27473 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:34.391119 27473 webserver.cc:460] Webserver started at http://127.24.130.190:41443/ using document root <none> and password file <none>
I20260806 02:50:34.391732 27473 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:34.391834 27473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:34.392099 27473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:34.394307 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "7dc26063ceb042bcb47cd680579ee549"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "f8de35712362fd0281bf50278e1f4ed9"
server_key_iv: "321c0a1664aa4d3d911ad0ed0c4af6a3"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.394968 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "7dc26063ceb042bcb47cd680579ee549"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "f8de35712362fd0281bf50278e1f4ed9"
server_key_iv: "321c0a1664aa4d3d911ad0ed0c4af6a3"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.398967 27473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:50:34.401497 27499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:34.402873 27473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:34.403029 27473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "7dc26063ceb042bcb47cd680579ee549"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "f8de35712362fd0281bf50278e1f4ed9"
server_key_iv: "321c0a1664aa4d3d911ad0ed0c4af6a3"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.403157 27473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:34.419924 27473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:34.423120 27473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:34.423362 27473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:34.432099 27473 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43523
I20260806 02:50:34.432119 27579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43523 every 8 connection(s)
I20260806 02:50:34.433451 27473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:34.436522 27580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:34.439744 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 27473
I20260806 02:50:34.439880 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:34.440155 25098 external_mini_cluster.cc:1428] Setting key d2f41f5b0948d728ab957a0da43564f3
I20260806 02:50:34.442957 27580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Bootstrap starting.
I20260806 02:50:34.446018 27580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:34.446973 27580 log.cc:824] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:34.449210 27580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: No bootstrap required, opened a new log
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:34.453207 27580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc26063ceb042bcb47cd680579ee549" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43523 } }
I20260806 02:50:34.453575 27580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:34.453660 27580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc26063ceb042bcb47cd680579ee549, State: Initialized, Role: FOLLOWER
I20260806 02:50:34.454210 27580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc26063ceb042bcb47cd680579ee549" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43523 } }
I20260806 02:50:34.454352 27580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:34.454402 27580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:34.454473 27580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:34.455528 27580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc26063ceb042bcb47cd680579ee549" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43523 } }
I20260806 02:50:34.455946 27580 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dc26063ceb042bcb47cd680579ee549; no voters:
I20260806 02:50:34.456312 27580 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:34.456413 27585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:34.456647 27585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [term 1 LEADER]: Becoming Leader. State: Replica: 7dc26063ceb042bcb47cd680579ee549, State: Running, Role: LEADER
I20260806 02:50:34.456833 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.441710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.24.130.190:43523)
0806 02:50:34.442191 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:34.442212 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:34.442259 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:50:34.443138 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:34.443162 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:34.443603 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:34.444116 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:34.444796 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.446354 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.447320 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.447939 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.448112 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.451184 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:34.451190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:34.451205 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:34.451236 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:34.453204 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.453779 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.453785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.453793 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.453947 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.454262 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.454266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.454268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.454892 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:34.455122 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:34.455451 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:50:34.455802 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":148,"threads_started":1}
I20260806 02:50:34.456991 27585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc26063ceb042bcb47cd680579ee549" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43523 } }
I20260806 02:50:34.457739 27580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:34.458305 27585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc26063ceb042bcb47cd680579ee549. Latest consensus state: current_term: 1 leader_uuid: "7dc26063ceb042bcb47cd680579ee549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc26063ceb042bcb47cd680579ee549" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43523 } } }
I20260806 02:50:34.458421 27585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:34.459120 27586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dc26063ceb042bcb47cd680579ee549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc26063ceb042bcb47cd680579ee549" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43523 } } }
I20260806 02:50:34.459234 27586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:34.459518 27595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:34.463153 27595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:34.468765 27595 catalog_manager.cc:1347] Generated new cluster ID: 34cd803e186f4d9c99eef24530b0d0d2
I20260806 02:50:34.468847 27595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:34.479214 27595 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:34.480325 27595 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:34.485127 27595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Generated new TSK 0
I20260806 02:50:34.485862 27595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:34.523068 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43523
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:34.642831 27620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:34.643185 27620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:34.643282 27620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:34.647403 27620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:34.647498 27620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:34.647572 27620 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:50:34.647617 27620 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:50:34.647696 27620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:34.653337 27620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:43523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:34.654539 27620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:34.655652 27620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:34.666049 27627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:34.666049 27628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:34.666049 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:34.666471 27620 server_base.cc:1034] running on GCE node
I20260806 02:50:34.667034 27620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:34.667743 27620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:34.668946 27620 hybrid_clock.cc:648] HybridClock initialized: now 1785984634668928 us; error 33 us; skew 500 ppm
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:34.675431 27620 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:34.676456 27620 webserver.cc:460] Webserver started at http://127.24.130.129:33645/ using document root <none> and password file <none>
I20260806 02:50:34.677073 27620 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:34.677143 27620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:34.677417 27620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:34.679755 27620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "086cef48f52043ca8c1347e5cc90ebfc"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "3143a72ffdc33649c79db35fa24c840d"
server_key_iv: "6303376302cacaab420d596682b8caab"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.680413 27620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "086cef48f52043ca8c1347e5cc90ebfc"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "3143a72ffdc33649c79db35fa24c840d"
server_key_iv: "6303376302cacaab420d596682b8caab"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.684716 27620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:34.687522 27641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:34.688884 27620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:50:34.689029 27620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "086cef48f52043ca8c1347e5cc90ebfc"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "3143a72ffdc33649c79db35fa24c840d"
server_key_iv: "6303376302cacaab420d596682b8caab"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.689168 27620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:34.709180 27620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:34.712797 27620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:34.712972 27620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:34.714433 27620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:34.714519 27620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:34.714591 27620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:34.714623 27620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:34.726219 27620 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:36397
I20260806 02:50:34.726831 27796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:36397 every 8 connection(s)
I20260806 02:50:34.727600 27620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:34.729761 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.715424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:46219 (local address 127.24.130.190:43523)
0806 02:50:34.715585 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:34.715590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:34.716501 (+ 911us) server_negotiation.cc:407] Connection header received
0806 02:50:34.717616 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:34.717620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:34.717693 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:34.717823 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:34.719519 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.720193 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.721470 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.721716 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.721836 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.725538 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:34.725543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:34.725545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:34.725580 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:34.727515 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.728151 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.728155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.728156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.728253 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.728684 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.728688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.728689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.728953 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:34.729042 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:34.729518 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 02:50:34.729636 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:50:34.730604 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.715839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.24.130.129:46219)
0806 02:50:34.716308 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:34.716372 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:34.717367 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:34.718069 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:34.718082 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:34.718579 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:34.719330 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.719351 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.720335 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.720339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.721323 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.721336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.721907 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.721912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.722105 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.722827 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:34.722845 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:34.725307 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:34.727661 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.727669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.727685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.728011 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.728380 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.728385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.728388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.728574 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.729055 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:34.729060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:34.729296 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 02:50:34.729465 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:50:34.729729 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":127,"threads_started":1}
I20260806 02:50:34.731700 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 27620
I20260806 02:50:34.731827 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:34.732159 25098 external_mini_cluster.cc:1428] Setting key 1b698d05d7e91c63edb799758866ae27
I20260806 02:50:34.732194 27798 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43523
I20260806 02:50:34.732362 27798 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:34.732998 27798 heartbeater.cc:511] Master 127.24.130.190:43523 requested a full tablet report, sending...
I20260806 02:50:34.734635 27522 ts_manager.cc:194] Registered new tserver with Master: 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397)
I20260806 02:50:34.736320 27522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:46219
I20260806 02:50:34.743112 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.734658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.24.130.129:43231)
0806 02:50:34.735036 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:34.735049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:34.735165 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:34.735427 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:34.735430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:34.735647 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:34.735749 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.735755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.736883 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.736887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.737659 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.737670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.738337 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.738342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.738436 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.739069 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:34.739080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:34.739428 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:34.741536 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.741539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.741541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.741752 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.742062 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.742064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.742065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.742146 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.742606 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:34.742608 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:34.742741 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 02:50:34.742831 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:50:34.742955 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":95,"threads_started":1}
I20260806 02:50:34.743266 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.734733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:43231 (local address 127.24.130.190:43523)
0806 02:50:34.734856 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:34.734861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:34.735125 (+ 264us) server_negotiation.cc:407] Connection header received
0806 02:50:34.735267 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:34.735270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:34.735320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:34.735410 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:34.735868 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.736716 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.737786 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.738196 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.738323 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.739572 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:34.739576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:34.739579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:34.739603 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:34.741395 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.741859 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.741863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.741865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.741962 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.742247 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.742252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.742255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.742510 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:34.742625 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:34.742933 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 02:50:34.743077 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:34.766366 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43523
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:34.885404 27816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:34.885702 27816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:34.885753 27816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:34.889858 27816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:34.889940 27816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:34.889981 27816 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:50:34.889997 27816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:50:34.890084 27816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:34.895784 27816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:43523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:34.896940 27816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:34.898006 27816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:34.907930 27822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:34.907925 27823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:34.907958 27825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:34.908363 27816 server_base.cc:1034] running on GCE node
I20260806 02:50:34.908803 27816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:34.909453 27816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:34.910614 27816 hybrid_clock.cc:648] HybridClock initialized: now 1785984634910590 us; error 36 us; skew 500 ppm
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:34.913559 27816 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:34.914499 27816 webserver.cc:460] Webserver started at http://127.24.130.130:39167/ using document root <none> and password file <none>
I20260806 02:50:34.915153 27816 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:34.915249 27816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:34.915477 27816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:34.917598 27816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "9f7b3f23babc4727bcf6452de3d61b98"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "d14d901de123f9cddaf9bfc69f1cf98a"
server_key_iv: "968a23acee6c099d4f2d5bc8fe020359"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.918224 27816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "9f7b3f23babc4727bcf6452de3d61b98"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "d14d901de123f9cddaf9bfc69f1cf98a"
server_key_iv: "968a23acee6c099d4f2d5bc8fe020359"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.922264 27816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:34.924880 27833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:34.926249 27816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:50:34.926404 27816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "9f7b3f23babc4727bcf6452de3d61b98"
format_stamp: "Formatted at 2026-08-06 02:50:34 on dist-test-slave-3kk6"
server_key: "d14d901de123f9cddaf9bfc69f1cf98a"
server_key_iv: "968a23acee6c099d4f2d5bc8fe020359"
server_key_version: "encryptionkey@0"
I20260806 02:50:34.926538 27816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:34.945690 27816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:34.949308 27816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:34.949560 27816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:34.951368 27816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:34.951452 27816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:34.951519 27816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:34.951557 27816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:50:34 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:34.963939 27816 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:38781
I20260806 02:50:34.964248 28003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:38781 every 8 connection(s)
I20260806 02:50:34.965014 27816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:34.967168 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.952145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:39147 (local address 127.24.130.190:43523)
0806 02:50:34.952333 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:34.952339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:34.953731 (+ 1392us) server_negotiation.cc:407] Connection header received
0806 02:50:34.955161 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:34.955164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:34.955229 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:34.955340 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:34.956789 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.957391 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.959159 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.959493 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.959627 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.962887 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:34.962891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:34.962894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:34.962932 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:34.964759 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.965437 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.965440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.965442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.965554 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.966004 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.966007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.966008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.966219 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:34.966309 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:34.966920 (+ 611us) server_negotiation.cc:299] Negotiation successful
0806 02:50:34.967040 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:50:34.968055 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.953031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.24.130.130:39147)
0806 02:50:34.953525 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:34.953583 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:34.954908 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:34.955534 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:34.955541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:34.955988 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:34.956613 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.956638 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.957524 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.957528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.958975 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.958990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.959690 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.959695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.959943 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.960836 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:34.960855 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:34.962605 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:34.964916 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.964923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.964939 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.965307 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.965669 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.965672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.965674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.965863 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.966325 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:34.966331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:34.966616 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 02:50:34.966845 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 02:50:34.967132 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":147,"threads_started":1}
I20260806 02:50:34.969403 28004 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43523
I20260806 02:50:34.969581 28004 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:34.970103 28004 heartbeater.cc:511] Master 127.24.130.190:43523 requested a full tablet report, sending...
I20260806 02:50:34.971282 27523 ts_manager.cc:194] Registered new tserver with Master: 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781)
I20260806 02:50:34.972074 27523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:39147
I20260806 02:50:34.975342 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 27816
I20260806 02:50:34.975484 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:34.975786 25098 external_mini_cluster.cc:1428] Setting key fb67ba37cb09d3e7f0d395ecb536d3a0
I20260806 02:50:34.980010 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.971691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.24.130.130:45787)
0806 02:50:34.971998 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:34.972013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:34.972104 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:34.972317 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:34.972340 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:34.972590 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:34.972704 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.972711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.973506 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.973511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.974077 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:34.974086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.974585 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.974588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:34.974742 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.975343 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:34.975358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:34.975794 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:34.977892 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.977896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.977898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.978198 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.978566 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:34.978573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:34.978578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:34.978723 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:34.979263 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:34.979269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:34.979414 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 02:50:34.979539 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 02:50:34.979740 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":124,"threads_started":1}
I20260806 02:50:34.980020 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:34.971771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:45787 (local address 127.24.130.190:43523)
0806 02:50:34.971906 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:34.971910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:34.972091 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:50:34.972175 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:34.972178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:34.972219 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:34.972309 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:34.972829 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.973363 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.974218 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:34.974456 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:34.974567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:34.976003 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:34.976006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:34.976008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:34.976042 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:34.977767 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.978331 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.978334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.978336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.978426 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:34.978902 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:34.978905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:34.978906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:34.979152 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:34.979251 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:34.979623 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 02:50:34.979786 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:35.005725 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43523
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:35.123888 28014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:35.124228 28014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:35.124320 28014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:35.128350 28014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:35.128448 28014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:35.128522 28014 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:50:35.128566 28014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:50:35.128636 28014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:35.134374 28014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:43523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:35.135524 28014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:35.136574 28014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:35.146345 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:35.146369 28019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:35.146349 28020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:35.146417 28014 server_base.cc:1034] running on GCE node
I20260806 02:50:35.147055 28014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:35.147710 28014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:35.148907 28014 hybrid_clock.cc:648] HybridClock initialized: now 1785984635148893 us; error 28 us; skew 500 ppm
Aug 06 02:50:35 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:35.151860 28014 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:35.152905 28014 webserver.cc:460] Webserver started at http://127.24.130.131:35403/ using document root <none> and password file <none>
I20260806 02:50:35.153496 28014 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:35.153594 28014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:35.153839 28014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:35.156126 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "e0732da08d29462a8b32d999b773f83e"
format_stamp: "Formatted at 2026-08-06 02:50:35 on dist-test-slave-3kk6"
server_key: "9e8451dbc2a7deba088f30708ca8cf3e"
server_key_iv: "f386d750e813d3f8caeac8279dc9ccc5"
server_key_version: "encryptionkey@0"
I20260806 02:50:35.156795 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "e0732da08d29462a8b32d999b773f83e"
format_stamp: "Formatted at 2026-08-06 02:50:35 on dist-test-slave-3kk6"
server_key: "9e8451dbc2a7deba088f30708ca8cf3e"
server_key_iv: "f386d750e813d3f8caeac8279dc9ccc5"
server_key_version: "encryptionkey@0"
I20260806 02:50:35.161166 28014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:35.163923 28034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:35.165143 28014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:35.165292 28014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "e0732da08d29462a8b32d999b773f83e"
format_stamp: "Formatted at 2026-08-06 02:50:35 on dist-test-slave-3kk6"
server_key: "9e8451dbc2a7deba088f30708ca8cf3e"
server_key_iv: "f386d750e813d3f8caeac8279dc9ccc5"
server_key_version: "encryptionkey@0"
I20260806 02:50:35.165457 28014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:35.183501 28014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:35.186619 28014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:35.186945 28014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:35.188583 28014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:35.188637 28014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:35.188678 28014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:35.188721 28014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:50:35 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:35.200814 28014 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:41251
I20260806 02:50:35.201603 28194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:41251 every 8 connection(s)
I20260806 02:50:35.202026 28014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:35.203603 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.189434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:34715 (local address 127.24.130.190:43523)
0806 02:50:35.189570 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.189573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.190684 (+ 1111us) server_negotiation.cc:407] Connection header received
0806 02:50:35.191832 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.191836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.191904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.191966 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:35.193618 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.194255 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.195533 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.195836 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.195967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.199378 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:35.199382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:35.199385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:35.199418 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:35.201219 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.201812 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.201815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.201817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.201904 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.202353 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.202356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.202357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.202610 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:35.202772 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.203360 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.203477 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:50:35.203766 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 28014
I20260806 02:50:35.203897 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:35.204218 25098 external_mini_cluster.cc:1428] Setting key b4ae7bf1e88df49022a51a5aa682e514
I20260806 02:50:35.204635 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.189997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.24.130.131:34715)
0806 02:50:35.190518 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.190564 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.191526 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.192472 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.192480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.192833 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:35.193436 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.193460 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.194400 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.194404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.195392 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.195402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.196028 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.196033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.196289 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.197074 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:35.197101 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:35.199160 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:35.201355 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:35.201362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:35.201379 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:35.201659 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:35.202011 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:35.202015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:35.202017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:35.202229 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:35.202821 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:35.202828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:35.203108 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.203262 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.203592 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":149,"threads_started":1}
I20260806 02:50:35.205971 28195 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43523
I20260806 02:50:35.206112 28195 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:35.206899 28195 heartbeater.cc:511] Master 127.24.130.190:43523 requested a full tablet report, sending...
I20260806 02:50:35.208040 27523 ts_manager.cc:194] Registered new tserver with Master: e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251)
I20260806 02:50:35.208891 27523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:34715
I20260806 02:50:35.215633 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.208449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:50107 (local address 127.24.130.190:43523)
0806 02:50:35.208579 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.208583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.208797 (+ 214us) server_negotiation.cc:407] Connection header received
0806 02:50:35.208864 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.208866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.208926 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.209042 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:35.209536 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.210041 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.210959 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.211176 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.211285 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.212531 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:35.212534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:35.212537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:35.212567 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:35.213875 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.214339 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.214344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.214346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.214445 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.214767 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.214771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.214773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.214973 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:35.215088 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.215362 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.215477 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:50:35.215644 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.208346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.24.130.131:50107)
0806 02:50:35.208660 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.208677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.208782 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.209063 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.209065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.209292 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:35.209415 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.209421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.210173 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.210176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.210839 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.210849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.211301 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.211304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.211437 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.212019 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:35.212031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:35.212380 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:35.214002 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:35.214005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:35.214007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:35.214221 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:35.214540 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:35.214545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:35.214547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:35.214640 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:35.215076 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:35.215079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:35.215184 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.215291 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.215462 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260806 02:50:35.218447 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:50:35 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:50:35 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:35.248325 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.239085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.24.130.190:43523)
0806 02:50:35.239224 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.239228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.239240 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:50:35.239282 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.239284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.239323 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.239420 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:35.240177 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.240729 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.241595 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.241822 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.241950 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.245154 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:35.245158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:35.245160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:35.245190 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:35.246507 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.246998 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.247002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.247006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.247094 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.247393 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.247396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.247398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.247599 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:35.247749 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.248017 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.248149 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:50:35.251441 27523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:35.254539 27523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:35.270083 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.263440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.24.130.130:38781)
0806 02:50:35.264627 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.264639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.264663 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:50:35.264751 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.264759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.265017 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.265225 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:35.265705 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.266543 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.268313 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.269210 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.269382 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.269549 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.269795 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.269909 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":137,"threads_started":1}
I20260806 02:50:35.270089 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.264251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.24.130.129:36397)
0806 02:50:35.264741 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.264747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.264771 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:50:35.264891 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.264896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.265091 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.265470 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:35.265785 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.266907 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.268682 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.269337 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.269557 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.269690 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.269860 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.269955 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":129,"threads_started":1}
I20260806 02:50:35.270320 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.264049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36397 (local address 127.0.0.1:37894)
0806 02:50:35.264661 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.264675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.264765 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.265242 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.265245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.265263 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:35.265672 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.265681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.267037 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.267040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.268545 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.268552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.269488 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.269491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.269660 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.269681 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.269832 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.270176 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":464,"thread_start_us":108,"threads_started":1}
I20260806 02:50:35.270504 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.263248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:38781 (local address 127.0.0.1:56152)
0806 02:50:35.263958 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.264030 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.264175 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.265215 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.265221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.265243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:35.265501 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.265510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.266684 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.266689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.268181 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.268191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.269351 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.269354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.269574 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.269620 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.269741 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.269945 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":77,"spinlock_wait_cycles":2304,"thread_start_us":102,"threads_started":1}
I20260806 02:50:35.270980 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.263466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.0.0.1:34908)
0806 02:50:35.264238 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.264253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.264345 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.265420 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.265425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.265443 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:35.265559 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.265569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.267075 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.267079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.268426 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.268439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.269485 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.269489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.270354 (+ 865us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.270462 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.270747 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.270828 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
I20260806 02:50:35.271411 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.264640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.24.130.131:41251)
0806 02:50:35.265020 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.265030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.265051 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:50:35.265119 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.265123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.265290 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.265456 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:35.265677 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.266948 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.268541 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.269325 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.269950 (+ 625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.270067 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.271152 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.271262 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":111,"threads_started":1}
I20260806 02:50:35.273378 27702 tablet_service.cc:1467] Processing CreateTablet for tablet ad9948bc9ef2428a959acbf6f21053bb (DEFAULT_TABLE table=test-table [id=cc51cdf472e84403b5117455c94b375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:35.273761 28102 tablet_service.cc:1467] Processing CreateTablet for tablet ad9948bc9ef2428a959acbf6f21053bb (DEFAULT_TABLE table=test-table [id=cc51cdf472e84403b5117455c94b375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:35.274314 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9948bc9ef2428a959acbf6f21053bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.274804 28102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9948bc9ef2428a959acbf6f21053bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.279454 27911 tablet_service.cc:1467] Processing CreateTablet for tablet ad9948bc9ef2428a959acbf6f21053bb (DEFAULT_TABLE table=test-table [id=cc51cdf472e84403b5117455c94b375a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:35.280594 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9948bc9ef2428a959acbf6f21053bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.281512 28227 tablet_bootstrap.cc:492] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Bootstrap starting.
I20260806 02:50:35.283021 28228 tablet_bootstrap.cc:492] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Bootstrap starting.
I20260806 02:50:35.284066 28227 tablet_bootstrap.cc:654] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.284981 28227 log.cc:824] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:35.285171 28228 tablet_bootstrap.cc:654] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.286028 28228 log.cc:824] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:35.287374 28227 tablet_bootstrap.cc:492] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: No bootstrap required, opened a new log
I20260806 02:50:35.287580 28227 ts_tablet_manager.cc:1397] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260806 02:50:35.288365 28228 tablet_bootstrap.cc:492] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: No bootstrap required, opened a new log
I20260806 02:50:35.288376 28232 tablet_bootstrap.cc:492] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Bootstrap starting.
I20260806 02:50:35.288545 28228 ts_tablet_manager.cc:1397] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:50:35.290082 28232 tablet_bootstrap.cc:654] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.290827 28232 log.cc:824] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:35.292670 28232 tablet_bootstrap.cc:492] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: No bootstrap required, opened a new log
I20260806 02:50:35.292472 28227 raft_consensus.cc:348] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.292541 28228 raft_consensus.cc:348] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.292764 28227 raft_consensus.cc:374] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.292764 28228 raft_consensus.cc:374] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.292829 28227 raft_consensus.cc:729] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cef48f52043ca8c1347e5cc90ebfc, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.292830 28228 raft_consensus.cc:729] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.292865 28232 ts_tablet_manager.cc:1397] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 02:50:35.293357 28228 consensus_queue.cc:260] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.293365 28227 consensus_queue.cc:260] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.294386 28228 ts_tablet_manager.cc:1428] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:35.294392 28227 ts_tablet_manager.cc:1428] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 02:50:35.294782 28195 heartbeater.cc:503] Master 127.24.130.190:43523 was elected leader, sending a full tablet report...
I20260806 02:50:35.295812 27798 heartbeater.cc:503] Master 127.24.130.190:43523 was elected leader, sending a full tablet report...
I20260806 02:50:35.296422 28232 raft_consensus.cc:348] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.296701 28232 raft_consensus.cc:374] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.296770 28232 raft_consensus.cc:729] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7b3f23babc4727bcf6452de3d61b98, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.297243 28232 consensus_queue.cc:260] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.298017 28004 heartbeater.cc:503] Master 127.24.130.190:43523 was elected leader, sending a full tablet report...
I20260806 02:50:35.298307 28232 ts_tablet_manager.cc:1428] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260806 02:50:35.356747 28235 raft_consensus.cc:482] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:35.357019 28235 raft_consensus.cc:504] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.358433 28235 leader_election.cc:288] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781), 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397)
I20260806 02:50:35.363276 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.359070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:55011 (local address 127.24.130.129:36397)
0806 02:50:35.359203 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.359210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.359435 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:50:35.359589 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.359591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.359633 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.359674 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:35.360130 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.360741 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.362094 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.362642 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.362807 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.362851 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.363054 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.363130 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:35.363306 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.358877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36397 (local address 127.24.130.131:55011)
0806 02:50:35.359321 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.359336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.359446 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.359841 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.359844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.359861 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:35.360013 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.360019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.360881 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.360885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.361961 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.361972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.362821 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.362825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.362923 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.362952 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.363043 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.363133 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":177,"threads_started":1}
I20260806 02:50:35.363607 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.358816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:38781 (local address 127.24.130.131:41315)
0806 02:50:35.359034 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.359052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.359171 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.359458 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.359463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.359483 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:35.359629 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.359635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.360567 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.360571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.362322 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.362333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.363166 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.363170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.363279 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.363313 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.363395 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.363473 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 02:50:35.363976 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.358878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:41315 (local address 127.24.130.130:38781)
0806 02:50:35.359036 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.359043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.359116 (+ 73us) server_negotiation.cc:407] Connection header received
0806 02:50:35.359283 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.359287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.359344 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.359430 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:35.359753 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.360457 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.362448 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.363052 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.363163 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.363210 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.363797 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.363885 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:50:35.364001 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9948bc9ef2428a959acbf6f21053bb" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc" is_pre_election: true
I20260806 02:50:35.364388 27728 raft_consensus.cc:2454] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 0.
I20260806 02:50:35.364544 27930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9948bc9ef2428a959acbf6f21053bb" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98" is_pre_election: true
I20260806 02:50:35.364836 27930 raft_consensus.cc:2454] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 0.
I20260806 02:50:35.364931 28037 leader_election.cc:302] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cef48f52043ca8c1347e5cc90ebfc, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:35.365280 28235 raft_consensus.cc:2781] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:35.365379 28235 raft_consensus.cc:482] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:35.365424 28235 raft_consensus.cc:3037] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.366530 28235 raft_consensus.cc:504] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.366956 28235 leader_election.cc:288] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 election: Requested vote from peers 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781), 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397)
I20260806 02:50:35.367298 27930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9948bc9ef2428a959acbf6f21053bb" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98"
I20260806 02:50:35.367437 27930 raft_consensus.cc:3037] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.367465 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9948bc9ef2428a959acbf6f21053bb" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc"
I20260806 02:50:35.367630 27728 raft_consensus.cc:3037] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.368605 27930 raft_consensus.cc:2454] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 1.
I20260806 02:50:35.368901 27728 raft_consensus.cc:2454] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 1.
I20260806 02:50:35.369005 28036 leader_election.cc:302] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7b3f23babc4727bcf6452de3d61b98, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:35.369236 28235 raft_consensus.cc:2781] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:35.369499 28235 raft_consensus.cc:686] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 1 LEADER]: Becoming Leader. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Running, Role: LEADER
I20260806 02:50:35.369872 28235 consensus_queue.cc:237] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } }
I20260806 02:50:35.373302 27523 catalog_manager.cc:5495] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e reported cstate change: term changed from 0 to 1, leader changed from <none> to e0732da08d29462a8b32d999b773f83e (127.24.130.131). New cstate: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:50:35 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984634, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:35.413298 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.404779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43523 (local address 127.0.0.1:60524)
0806 02:50:35.405128 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.405143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.405262 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.405518 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.405521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.405948 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:35.406050 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.406057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.407001 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.407004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.407572 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.407580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.408102 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.408106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.408178 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.408836 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:35.408849 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:35.410310 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:35.411879 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:35.411888 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:35.411893 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:35.412123 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:35.412373 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:35.412375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:35.412376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:35.412454 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:35.412792 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:35.412797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:35.412902 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.412993 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.413117 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":149,"threads_started":1}
I20260806 02:50:35.413300 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.404831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.24.130.190:43523)
0806 02:50:35.404991 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.404996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.405232 (+ 236us) server_negotiation.cc:407] Connection header received
0806 02:50:35.405332 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.405335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.405376 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.405463 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:35.406212 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.406884 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.407730 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.407967 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.408060 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.410452 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:35.410455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:35.410457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:35.410478 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:35.411780 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.412212 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.412214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.412216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.412289 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.412531 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.412534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.412535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.412706 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:35.412779 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.413052 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.413176 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":61}
I20260806 02:50:35.417743 27523 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:60524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:50:35.418430 27523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:35.423523 27911 tablet_service.cc:1467] Processing CreateTablet for tablet fb32936e1cf546a7bdefd88597dd6da4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8ff135d491649f985b52e514e03cdbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:50:35.423369 27702 tablet_service.cc:1467] Processing CreateTablet for tablet fb32936e1cf546a7bdefd88597dd6da4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8ff135d491649f985b52e514e03cdbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:50:35.423489 28102 tablet_service.cc:1467] Processing CreateTablet for tablet fb32936e1cf546a7bdefd88597dd6da4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8ff135d491649f985b52e514e03cdbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:50:35.424183 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb32936e1cf546a7bdefd88597dd6da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.424183 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb32936e1cf546a7bdefd88597dd6da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.424222 28102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb32936e1cf546a7bdefd88597dd6da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.427196 28228 tablet_bootstrap.cc:492] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: Bootstrap starting.
I20260806 02:50:35.427553 28232 tablet_bootstrap.cc:492] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: Bootstrap starting.
I20260806 02:50:35.428166 28228 tablet_bootstrap.cc:654] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.428398 28232 tablet_bootstrap.cc:654] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.429126 28227 tablet_bootstrap.cc:492] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: Bootstrap starting.
I20260806 02:50:35.429432 28228 tablet_bootstrap.cc:492] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: No bootstrap required, opened a new log
I20260806 02:50:35.429508 28228 ts_tablet_manager.cc:1397] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:50:35.429698 28232 tablet_bootstrap.cc:492] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: No bootstrap required, opened a new log
I20260806 02:50:35.429759 28232 ts_tablet_manager.cc:1397] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 02:50:35.430001 28228 raft_consensus.cc:348] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.430151 28227 tablet_bootstrap.cc:654] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.430142 28228 raft_consensus.cc:374] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.430183 28228 raft_consensus.cc:729] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.430162 28232 raft_consensus.cc:348] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.430248 28232 raft_consensus.cc:374] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.430294 28232 raft_consensus.cc:729] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7b3f23babc4727bcf6452de3d61b98, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.430338 28228 consensus_queue.cc:260] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.430423 28232 consensus_queue.cc:260] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.430752 28232 ts_tablet_manager.cc:1428] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:35.430727 28235 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.430720 28236 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.430850 28235 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:35.430861 28236 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:35.431131 28228 ts_tablet_manager.cc:1428] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:35.431944 28227 tablet_bootstrap.cc:492] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: No bootstrap required, opened a new log
I20260806 02:50:35.432025 28227 ts_tablet_manager.cc:1397] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:50:35.432463 28227 raft_consensus.cc:348] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.432564 28227 raft_consensus.cc:374] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.432590 28227 raft_consensus.cc:729] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cef48f52043ca8c1347e5cc90ebfc, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.432706 28227 consensus_queue.cc:260] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.432960 28227 ts_tablet_manager.cc:1428] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:50:35.432973 28234 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.433142 28234 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 02:50:35.453246 28196 tablet.cc:2369] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:35.466344 28005 tablet.cc:2369] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:35.479117 27799 tablet.cc:2369] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:35.594053 28235 raft_consensus.cc:482] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:35.594228 28235 raft_consensus.cc:504] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.594790 28235 leader_election.cc:288] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397), 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781)
I20260806 02:50:35.595343 27930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb32936e1cf546a7bdefd88597dd6da4" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98" is_pre_election: true
I20260806 02:50:35.595379 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb32936e1cf546a7bdefd88597dd6da4" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc" is_pre_election: true
I20260806 02:50:35.595539 27930 raft_consensus.cc:2454] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 0.
I20260806 02:50:35.595546 27728 raft_consensus.cc:2454] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 0.
I20260806 02:50:35.595880 28036 leader_election.cc:302] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7b3f23babc4727bcf6452de3d61b98, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:35.596099 28235 raft_consensus.cc:2781] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:35.596164 28235 raft_consensus.cc:482] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:35.596194 28235 raft_consensus.cc:3037] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.597291 28235 raft_consensus.cc:504] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.597688 28235 leader_election.cc:288] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 election: Requested vote from peers 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397), 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781)
I20260806 02:50:35.598148 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb32936e1cf546a7bdefd88597dd6da4" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc"
I20260806 02:50:35.598148 27930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb32936e1cf546a7bdefd88597dd6da4" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98"
I20260806 02:50:35.598308 27728 raft_consensus.cc:3037] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.598308 27930 raft_consensus.cc:3037] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.599602 27930 raft_consensus.cc:2454] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 1.
I20260806 02:50:35.599596 27728 raft_consensus.cc:2454] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 1.
I20260806 02:50:35.599951 28036 leader_election.cc:302] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7b3f23babc4727bcf6452de3d61b98, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:35.600148 28235 raft_consensus.cc:2781] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:35.600258 28235 raft_consensus.cc:686] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [term 1 LEADER]: Becoming Leader. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Running, Role: LEADER
I20260806 02:50:35.600417 28235 consensus_queue.cc:237] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.601486 28240 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: TxnStatusTablet state changed. Reason: New leader e0732da08d29462a8b32d999b773f83e. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.601603 28240 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:50:35.601995 28258 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:50:35.602257 28258 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:50:35.602567 27522 catalog_manager.cc:5495] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e reported cstate change: term changed from 0 to 1, leader changed from <none> to e0732da08d29462a8b32d999b773f83e (127.24.130.131). New cstate: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } health_report { overall_health: HEALTHY } } }
I20260806 02:50:35.622540 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.618604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.24.130.131:41251)
0806 02:50:35.618750 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.618754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.618768 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:35.618923 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.618925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.618981 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.619064 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:35.619366 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.619863 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.621086 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.621291 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.621374 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.621716 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:35.622144 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:35.622239 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.622369 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.622424 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:50:35.622579 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.618472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.0.0.1:34922)
0806 02:50:35.618624 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.618641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.618773 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.619108 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.619111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.619119 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:35.619249 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.619256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.619980 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.619983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.620964 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.620974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.621399 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.621403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.621505 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.621562 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:35.622246 (+ 684us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:35.622263 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.622337 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.622430 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 02:50:35.628036 27728 raft_consensus.cc:1261] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Refusing update from remote peer e0732da08d29462a8b32d999b773f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:35.628036 27930 raft_consensus.cc:1261] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Refusing update from remote peer e0732da08d29462a8b32d999b773f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:35.628789 28235 consensus_queue.cc:1035] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:35.628995 28240 consensus_queue.cc:1035] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:35.633482 28236 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: TxnStatusTablet state changed. Reason: New leader e0732da08d29462a8b32d999b773f83e. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.633458 28234 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: TxnStatusTablet state changed. Reason: New leader e0732da08d29462a8b32d999b773f83e. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.633613 28234 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:35.633623 28236 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:35.637295 28240 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.637441 28240 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:50:35.637609 28236 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.637725 28236 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P 9f7b3f23babc4727bcf6452de3d61b98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:35.639930 28264 mvcc.cc:205] Tried to move back new op lower bound from 7315393067524714496 to 7315393067421155328. Current Snapshot: MvccSnapshot[applied={T|T < 7315393067524714496}]
I20260806 02:50:35.645196 28235 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.645340 28235 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P e0732da08d29462a8b32d999b773f83e: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:50:35.646360 28234 tablet_replica.cc:439] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } } }
I20260806 02:50:35.646483 28234 tablet_replica.cc:442] T fb32936e1cf546a7bdefd88597dd6da4 P 086cef48f52043ca8c1347e5cc90ebfc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:50:35.647119 28263 mvcc.cc:205] Tried to move back new op lower bound from 7315393067524714496 to 7315393067421155328. Current Snapshot: MvccSnapshot[applied={T|T < 7315393067524714496}]
I20260806 02:50:35.647313 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.643202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.24.130.131:41251)
0806 02:50:35.643326 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.643330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.643344 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:35.643448 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.643452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.643498 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.643591 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:35.643854 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.644482 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.645673 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.645984 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.646089 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.646474 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:35.646556 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:35.646623 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.647128 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.647184 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 02:50:35.660338 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.656263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.24.130.131:38475)
0806 02:50:35.656437 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:35.656453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:35.656535 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:35.656797 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:35.656799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:35.656804 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:35.656914 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.656918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.657958 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.657961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.658978 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:35.658990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.659505 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.659509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:35.659611 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.659672 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:35.660023 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:35.660030 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:50:35.660091 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 02:50:35.660194 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:50:35.660370 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.656360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38475 (local address 127.24.130.131:41251)
0806 02:50:35.656482 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.656487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.656504 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:50:35.656636 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.656639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.656690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.656769 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:35.657010 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.657796 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.659143 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.659388 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.659474 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.659854 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:35.659942 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:35.660006 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.660135 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.660207 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260806 02:50:35.667615 27728 raft_consensus.cc:1261] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Refusing update from remote peer e0732da08d29462a8b32d999b773f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:35.667678 27930 raft_consensus.cc:1261] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Refusing update from remote peer e0732da08d29462a8b32d999b773f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:35.668092 28272 consensus_queue.cc:1035] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:35.668287 28235 consensus_queue.cc:1035] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:35.672345 28261 mvcc.cc:205] Tried to move back new op lower bound from 7315393067691106304 to 7315393066478039040. Current Snapshot: MvccSnapshot[applied={T|T < 7315393067691106304}]
I20260806 02:50:35.715945 27522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:35.716162 27522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:35.719318 27522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Sending DeleteTablet for 3 replicas of tablet ad9948bc9ef2428a959acbf6f21053bb
I20260806 02:50:35.720166 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9948bc9ef2428a959acbf6f21053bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:35 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 02:50:35.720309 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9948bc9ef2428a959acbf6f21053bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:35 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:37894
I20260806 02:50:35.720466 28101 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9948bc9ef2428a959acbf6f21053bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:35 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:34908
I20260806 02:50:35.720794 28282 tablet_replica.cc:330] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: stopping tablet replica
I20260806 02:50:35.720872 28283 tablet_replica.cc:330] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: stopping tablet replica
I20260806 02:50:35.721218 28282 raft_consensus.cc:2229] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:35.721218 28283 raft_consensus.cc:2229] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:35.721467 28283 raft_consensus.cc:2258] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:35.721493 28282 raft_consensus.cc:2258] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:35.721581 27522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:35.722052 27522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:35.723290 28282 ts_tablet_manager.cc:1905] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:35.723898 28283 ts_tablet_manager.cc:1905] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:35.727028 28282 ts_tablet_manager.cc:1918] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:35.727227 28282 log.cc:1196] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/ad9948bc9ef2428a959acbf6f21053bb
I20260806 02:50:35.727660 28282 ts_tablet_manager.cc:1939] T ad9948bc9ef2428a959acbf6f21053bb P 9f7b3f23babc4727bcf6452de3d61b98: Deleting consensus metadata
I20260806 02:50:35.728024 27702 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab27e34f03b456788f3b5516f86eeda (DEFAULT_TABLE table=test-table [id=234addbe2e964bffb68854b4addbcef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:35.728332 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab27e34f03b456788f3b5516f86eeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.730824 28227 tablet_bootstrap.cc:492] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Bootstrap starting.
I20260806 02:50:35.731417 28284 tablet_replica.cc:330] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: stopping tablet replica
I20260806 02:50:35.731750 28101 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab27e34f03b456788f3b5516f86eeda (DEFAULT_TABLE table=test-table [id=234addbe2e964bffb68854b4addbcef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:35.731887 28227 tablet_bootstrap.cc:654] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.731824 28284 raft_consensus.cc:2229] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:35.732082 28101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab27e34f03b456788f3b5516f86eeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.732524 28284 raft_consensus.cc:2258] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:35.734202 28284 ts_tablet_manager.cc:1905] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:35.734151 27911 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab27e34f03b456788f3b5516f86eeda (DEFAULT_TABLE table=test-table [id=234addbe2e964bffb68854b4addbcef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:35.734445 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab27e34f03b456788f3b5516f86eeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:35.732182 27504 catalog_manager.cc:4841] TS 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781): tablet ad9948bc9ef2428a959acbf6f21053bb (table test-table [id=cc51cdf472e84403b5117455c94b375a]) successfully deleted
I20260806 02:50:35.735541 28228 tablet_bootstrap.cc:492] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Bootstrap starting.
I20260806 02:50:35.736554 28228 tablet_bootstrap.cc:654] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.737358 28232 tablet_bootstrap.cc:492] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Bootstrap starting.
I20260806 02:50:35.737649 28284 ts_tablet_manager.cc:1918] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:35.737855 28284 log.cc:1196] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/ad9948bc9ef2428a959acbf6f21053bb
I20260806 02:50:35.738337 28284 ts_tablet_manager.cc:1939] T ad9948bc9ef2428a959acbf6f21053bb P e0732da08d29462a8b32d999b773f83e: Deleting consensus metadata
I20260806 02:50:35.738435 28232 tablet_bootstrap.cc:654] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.739804 27505 catalog_manager.cc:4841] TS e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251): tablet ad9948bc9ef2428a959acbf6f21053bb (table test-table [id=cc51cdf472e84403b5117455c94b375a]) successfully deleted
I20260806 02:50:35.740008 28232 tablet_bootstrap.cc:492] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: No bootstrap required, opened a new log
I20260806 02:50:35.740108 28232 ts_tablet_manager.cc:1397] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:50:35.740590 28232 raft_consensus.cc:348] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.740726 28232 raft_consensus.cc:374] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.740772 28232 raft_consensus.cc:729] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7b3f23babc4727bcf6452de3d61b98, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.740911 28232 consensus_queue.cc:260] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.741197 28232 ts_tablet_manager.cc:1428] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 02:50:35.744421 28283 ts_tablet_manager.cc:1918] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:35.744556 28283 log.cc:1196] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/ad9948bc9ef2428a959acbf6f21053bb
I20260806 02:50:35.744982 28283 ts_tablet_manager.cc:1939] T ad9948bc9ef2428a959acbf6f21053bb P 086cef48f52043ca8c1347e5cc90ebfc: Deleting consensus metadata
I20260806 02:50:35.746414 27505 catalog_manager.cc:4841] TS 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397): tablet ad9948bc9ef2428a959acbf6f21053bb (table test-table [id=cc51cdf472e84403b5117455c94b375a]) successfully deleted
I20260806 02:50:35.747246 28228 tablet_bootstrap.cc:492] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: No bootstrap required, opened a new log
I20260806 02:50:35.747366 28228 ts_tablet_manager.cc:1397] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260806 02:50:35.747947 28228 raft_consensus.cc:348] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.748098 28228 raft_consensus.cc:374] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.748142 28228 raft_consensus.cc:729] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.748294 28228 consensus_queue.cc:260] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.748463 28227 tablet_bootstrap.cc:492] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: No bootstrap required, opened a new log
I20260806 02:50:35.748559 28227 ts_tablet_manager.cc:1397] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.001s
I20260806 02:50:35.748613 28228 ts_tablet_manager.cc:1428] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:50:35.749003 28227 raft_consensus.cc:348] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.749122 28227 raft_consensus.cc:374] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.749158 28227 raft_consensus.cc:729] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cef48f52043ca8c1347e5cc90ebfc, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.749275 28227 consensus_queue.cc:260] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.749511 28227 ts_tablet_manager.cc:1428] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:50:35.995492 28240 raft_consensus.cc:482] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:35.995677 28240 raft_consensus.cc:504] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.996237 28240 leader_election.cc:288] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397), 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781)
I20260806 02:50:35.996866 27935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab27e34f03b456788f3b5516f86eeda" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98" is_pre_election: true
I20260806 02:50:35.996863 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab27e34f03b456788f3b5516f86eeda" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc" is_pre_election: true
I20260806 02:50:35.997046 27728 raft_consensus.cc:2454] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 0.
I20260806 02:50:35.997046 27935 raft_consensus.cc:2454] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 0.
I20260806 02:50:35.997427 28036 leader_election.cc:302] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7b3f23babc4727bcf6452de3d61b98, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:35.997637 28240 raft_consensus.cc:2781] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:35.997705 28240 raft_consensus.cc:482] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:35.997735 28240 raft_consensus.cc:3037] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.998921 28240 raft_consensus.cc:504] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:35.999301 28240 leader_election.cc:288] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 election: Requested vote from peers 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397), 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781)
I20260806 02:50:35.999696 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab27e34f03b456788f3b5516f86eeda" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc"
I20260806 02:50:35.999773 27935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab27e34f03b456788f3b5516f86eeda" candidate_uuid: "e0732da08d29462a8b32d999b773f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98"
I20260806 02:50:35.999856 27728 raft_consensus.cc:3037] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.999897 27935 raft_consensus.cc:3037] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:36.001086 27935 raft_consensus.cc:2454] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 1.
I20260806 02:50:36.001086 27728 raft_consensus.cc:2454] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0732da08d29462a8b32d999b773f83e in term 1.
I20260806 02:50:36.001434 28036 leader_election.cc:302] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7b3f23babc4727bcf6452de3d61b98, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:36.001643 28240 raft_consensus.cc:2781] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:36.001745 28240 raft_consensus.cc:686] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 1 LEADER]: Becoming Leader. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Running, Role: LEADER
I20260806 02:50:36.001924 28240 consensus_queue.cc:237] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:36.003787 27521 catalog_manager.cc:5495] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e reported cstate change: term changed from 0 to 1, leader changed from <none> to e0732da08d29462a8b32d999b773f83e (127.24.130.131). New cstate: current_term: 1 leader_uuid: "e0732da08d29462a8b32d999b773f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } health_report { overall_health: HEALTHY } } }
I20260806 02:50:36.050489 27728 raft_consensus.cc:1261] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Refusing update from remote peer e0732da08d29462a8b32d999b773f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:36.050546 27935 raft_consensus.cc:1261] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Refusing update from remote peer e0732da08d29462a8b32d999b773f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:36.050985 28235 consensus_queue.cc:1035] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:36.051159 28272 consensus_queue.cc:1035] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:50:36.065572 28261 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:50:37.621507 27522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:37.621685 27522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:37.624409 27522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Sending DeleteTablet for 3 replicas of tablet 3ab27e34f03b456788f3b5516f86eeda
I20260806 02:50:37.625098 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab27e34f03b456788f3b5516f86eeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:37894
I20260806 02:50:37.625120 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab27e34f03b456788f3b5516f86eeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 02:50:37.625120 28101 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab27e34f03b456788f3b5516f86eeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:34908
I20260806 02:50:37.625545 28328 tablet_replica.cc:330] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: stopping tablet replica
I20260806 02:50:37.625568 28329 tablet_replica.cc:330] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: stopping tablet replica
I20260806 02:50:37.625700 28329 raft_consensus.cc:2229] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:37.625715 28328 raft_consensus.cc:2229] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:37.625842 28329 raft_consensus.cc:2258] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.625849 28328 raft_consensus.cc:2258] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.626112 28327 tablet_replica.cc:330] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: stopping tablet replica
I20260806 02:50:37.626273 28327 raft_consensus.cc:2229] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:37.626523 28327 raft_consensus.cc:2258] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.626773 28328 ts_tablet_manager.cc:1905] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:37.626773 28329 ts_tablet_manager.cc:1905] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:37.627264 28327 ts_tablet_manager.cc:1905] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:37.629567 28328 ts_tablet_manager.cc:1918] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:50:37.629667 28328 log.cc:1196] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/3ab27e34f03b456788f3b5516f86eeda
I20260806 02:50:37.629858 28329 ts_tablet_manager.cc:1918] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:50:37.629995 28328 ts_tablet_manager.cc:1939] T 3ab27e34f03b456788f3b5516f86eeda P 9f7b3f23babc4727bcf6452de3d61b98: Deleting consensus metadata
I20260806 02:50:37.629995 28329 log.cc:1196] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/3ab27e34f03b456788f3b5516f86eeda
I20260806 02:50:37.630486 28329 ts_tablet_manager.cc:1939] T 3ab27e34f03b456788f3b5516f86eeda P 086cef48f52043ca8c1347e5cc90ebfc: Deleting consensus metadata
I20260806 02:50:37.630673 28327 ts_tablet_manager.cc:1918] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:50:37.630769 28327 log.cc:1196] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/3ab27e34f03b456788f3b5516f86eeda
I20260806 02:50:37.630936 27504 catalog_manager.cc:4841] TS 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781): tablet 3ab27e34f03b456788f3b5516f86eeda (table test-table [id=234addbe2e964bffb68854b4addbcef6]) successfully deleted
I20260806 02:50:37.631091 28327 ts_tablet_manager.cc:1939] T 3ab27e34f03b456788f3b5516f86eeda P e0732da08d29462a8b32d999b773f83e: Deleting consensus metadata
I20260806 02:50:37.631273 27505 catalog_manager.cc:4841] TS 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397): tablet 3ab27e34f03b456788f3b5516f86eeda (table test-table [id=234addbe2e964bffb68854b4addbcef6]) successfully deleted
I20260806 02:50:37.632028 27505 catalog_manager.cc:4841] TS e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251): tablet 3ab27e34f03b456788f3b5516f86eeda (table test-table [id=234addbe2e964bffb68854b4addbcef6]) successfully deleted
Aug 06 02:50:37 dist-test-slave-3kk6 krb5kdc[27456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:50:37 dist-test-slave-3kk6 krb5kdc[27456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:37.658298 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.648590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.24.130.190:43523)
0806 02:50:37.648927 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.648938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.649033 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:50:37.649246 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.649249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.649302 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.649407 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:37.650206 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.650866 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.651757 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.651990 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.652103 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.654900 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:37.654904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:37.654906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:37.654935 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:37.656402 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.656881 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.656885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.656887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.656973 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.657286 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.657290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.657291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.657500 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:37.657684 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.657960 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.658096 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":122,"threads_started":1}
I20260806 02:50:37.661441 27522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:37.661937 27522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:37.667397 27702 tablet_service.cc:1467] Processing CreateTablet for tablet 327da5a492354381a919fe3b10f5cc6d (DEFAULT_TABLE table=test-table [id=9beb5370bb54405e854ac184601b6b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:37.667497 27911 tablet_service.cc:1467] Processing CreateTablet for tablet 327da5a492354381a919fe3b10f5cc6d (DEFAULT_TABLE table=test-table [id=9beb5370bb54405e854ac184601b6b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:37.667521 28101 tablet_service.cc:1467] Processing CreateTablet for tablet 327da5a492354381a919fe3b10f5cc6d (DEFAULT_TABLE table=test-table [id=9beb5370bb54405e854ac184601b6b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:37.667757 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327da5a492354381a919fe3b10f5cc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:37.667757 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327da5a492354381a919fe3b10f5cc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:37.667810 28101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327da5a492354381a919fe3b10f5cc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:37.670418 28341 tablet_bootstrap.cc:492] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Bootstrap starting.
I20260806 02:50:37.670644 28342 tablet_bootstrap.cc:492] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Bootstrap starting.
I20260806 02:50:37.670786 28340 tablet_bootstrap.cc:492] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Bootstrap starting.
I20260806 02:50:37.671759 28342 tablet_bootstrap.cc:654] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:37.672163 28340 tablet_bootstrap.cc:654] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:37.673184 28342 tablet_bootstrap.cc:492] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: No bootstrap required, opened a new log
I20260806 02:50:37.673295 28342 ts_tablet_manager.cc:1397] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:50:37.673777 28341 tablet_bootstrap.cc:654] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:37.673746 28342 raft_consensus.cc:348] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.673892 28342 raft_consensus.cc:374] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:37.673940 28342 raft_consensus.cc:729] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Initialized, Role: FOLLOWER
I20260806 02:50:37.674111 28342 consensus_queue.cc:260] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.674443 28342 ts_tablet_manager.cc:1428] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:50:37.675316 28340 tablet_bootstrap.cc:492] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: No bootstrap required, opened a new log
I20260806 02:50:37.675433 28340 ts_tablet_manager.cc:1397] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:50:37.675649 28341 tablet_bootstrap.cc:492] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: No bootstrap required, opened a new log
I20260806 02:50:37.675745 28341 ts_tablet_manager.cc:1397] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:50:37.675959 28340 raft_consensus.cc:348] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.676083 28340 raft_consensus.cc:374] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:37.676118 28340 raft_consensus.cc:729] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7b3f23babc4727bcf6452de3d61b98, State: Initialized, Role: FOLLOWER
I20260806 02:50:37.676244 28341 raft_consensus.cc:348] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.676352 28341 raft_consensus.cc:374] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:37.676297 28340 consensus_queue.cc:260] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.676384 28341 raft_consensus.cc:729] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cef48f52043ca8c1347e5cc90ebfc, State: Initialized, Role: FOLLOWER
I20260806 02:50:37.676549 28341 consensus_queue.cc:260] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.676743 28340 ts_tablet_manager.cc:1428] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:50:37.676978 28341 ts_tablet_manager.cc:1428] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:50:37.871472 28352 raft_consensus.cc:482] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:37.871634 28352 raft_consensus.cc:504] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.872936 28352 leader_election.cc:288] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781), e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251)
I20260806 02:50:37.877396 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.873343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:42993 (local address 127.24.130.131:41251)
0806 02:50:37.873819 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.873829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.873846 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:50:37.874004 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.874007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.874057 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.874124 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.874482 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.875090 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.876235 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.876655 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.876756 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.876800 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.877143 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.877241 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
I20260806 02:50:37.877398 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.873267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.24.130.129:42993)
0806 02:50:37.873699 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.873715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.873878 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.874165 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.874171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.874188 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.874359 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.874367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.875206 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.875210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.876109 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.876120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.876901 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.876906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.877007 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.877032 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.877113 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.877192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":71,"spinlock_wait_cycles":46720,"thread_start_us":125,"threads_started":1}
I20260806 02:50:37.877427 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.873192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:38781 (local address 127.24.130.129:51573)
0806 02:50:37.873688 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.873708 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.873884 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.874214 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.874217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.874242 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.874376 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.874383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.875259 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.875261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.876143 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.876153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.876990 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.876993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.877116 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.877142 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.877220 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.877283 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":4,"spinlock_wait_cycles":3584,"thread_start_us":164,"threads_started":1}
I20260806 02:50:37.877493 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.873300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:51573 (local address 127.24.130.130:38781)
0806 02:50:37.873702 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.873715 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.873801 (+ 86us) server_negotiation.cc:407] Connection header received
0806 02:50:37.874033 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.874036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.874075 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.874182 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.874512 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.875101 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.876304 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.876841 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.876995 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.877048 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.877258 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.877335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260806 02:50:37.877964 27930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327da5a492354381a919fe3b10f5cc6d" candidate_uuid: "086cef48f52043ca8c1347e5cc90ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98" is_pre_election: true
I20260806 02:50:37.878114 27930 raft_consensus.cc:2454] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cef48f52043ca8c1347e5cc90ebfc in term 0.
I20260806 02:50:37.878576 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327da5a492354381a919fe3b10f5cc6d" candidate_uuid: "086cef48f52043ca8c1347e5cc90ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0732da08d29462a8b32d999b773f83e" is_pre_election: true
I20260806 02:50:37.878575 27643 leader_election.cc:302] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cef48f52043ca8c1347e5cc90ebfc, 9f7b3f23babc4727bcf6452de3d61b98; no voters:
I20260806 02:50:37.878854 28129 raft_consensus.cc:2454] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cef48f52043ca8c1347e5cc90ebfc in term 0.
I20260806 02:50:37.878928 28352 raft_consensus.cc:2781] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:37.879019 28352 raft_consensus.cc:482] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:37.879069 28352 raft_consensus.cc:3037] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:37.880177 28352 raft_consensus.cc:504] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.880585 28352 leader_election.cc:288] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [CANDIDATE]: Term 1 election: Requested vote from peers 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781), e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251)
I20260806 02:50:37.880919 27930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327da5a492354381a919fe3b10f5cc6d" candidate_uuid: "086cef48f52043ca8c1347e5cc90ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7b3f23babc4727bcf6452de3d61b98"
I20260806 02:50:37.880968 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327da5a492354381a919fe3b10f5cc6d" candidate_uuid: "086cef48f52043ca8c1347e5cc90ebfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0732da08d29462a8b32d999b773f83e"
I20260806 02:50:37.881067 27930 raft_consensus.cc:3037] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:37.881081 28129 raft_consensus.cc:3037] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:37.882012 28129 raft_consensus.cc:2454] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cef48f52043ca8c1347e5cc90ebfc in term 1.
I20260806 02:50:37.882053 27930 raft_consensus.cc:2454] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cef48f52043ca8c1347e5cc90ebfc in term 1.
I20260806 02:50:37.882295 27644 leader_election.cc:302] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cef48f52043ca8c1347e5cc90ebfc, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:37.882484 28352 raft_consensus.cc:2781] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:37.882776 28352 raft_consensus.cc:686] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 1 LEADER]: Becoming Leader. State: Replica: 086cef48f52043ca8c1347e5cc90ebfc, State: Running, Role: LEADER
I20260806 02:50:37.883066 28352 consensus_queue.cc:237] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:37.885391 27521 catalog_manager.cc:5495] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129). New cstate: current_term: 1 leader_uuid: "086cef48f52043ca8c1347e5cc90ebfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:37.922281 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.918365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.24.130.129:36397)
0806 02:50:37.918708 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.918716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.918730 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:37.918816 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.918818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.918872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.919004 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:37.919284 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.919835 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.920711 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.920972 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.921088 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.921385 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:37.921856 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:37.921994 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.922056 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.922113 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260806 02:50:37.933262 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.929312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.24.130.129:48653)
0806 02:50:37.929693 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.929708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.929882 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.930169 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.930172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.930179 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:37.930329 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.930335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.931188 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.931193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.931951 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.931961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.932437 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.932440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.932540 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.932589 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:37.932943 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:37.932953 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.933018 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.933100 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":135,"threads_started":1}
I20260806 02:50:37.933246 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.929513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:48653 (local address 127.24.130.131:41251)
0806 02:50:37.929666 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.929671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.929804 (+ 133us) server_negotiation.cc:407] Connection header received
0806 02:50:37.930001 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.930009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.930053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.930158 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:37.930436 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.931065 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.932081 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.932304 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.932421 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.932738 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:37.932822 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:37.932930 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.933028 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.933085 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:50:37.940037 27930 raft_consensus.cc:1261] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Refusing update from remote peer 086cef48f52043ca8c1347e5cc90ebfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:37.940156 28129 raft_consensus.cc:1261] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Refusing update from remote peer 086cef48f52043ca8c1347e5cc90ebfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:37.940563 28359 consensus_queue.cc:1035] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:37.941216 28359 consensus_queue.cc:1035] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:37.960457 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.956774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:36857 (local address 127.24.130.129:36397)
0806 02:50:37.956924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.956930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.957146 (+ 216us) server_negotiation.cc:407] Connection header received
0806 02:50:37.957292 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.957294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.957351 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.957469 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:37.957714 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.958240 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.959298 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.959596 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.959713 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.959978 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:37.960064 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:37.960145 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.960236 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.960296 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:50:37.960458 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.956683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36397 (local address 127.24.130.131:36857)
0806 02:50:37.957058 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.957070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.957191 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.957478 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.957480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.957488 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:37.957599 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.957605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.958393 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.958400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.959164 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.959175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.959713 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.959716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.959808 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.959846 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:37.960149 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:37.960158 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.960225 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.960314 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":107,"threads_started":1}
I20260806 02:50:37.991308 27521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:37.991519 27521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:37.994138 27521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Sending DeleteTablet for 3 replicas of tablet 327da5a492354381a919fe3b10f5cc6d
I20260806 02:50:37.994876 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet 327da5a492354381a919fe3b10f5cc6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 02:50:37.995159 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet 327da5a492354381a919fe3b10f5cc6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:37894
I20260806 02:50:37.995388 28328 tablet_replica.cc:330] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: stopping tablet replica
I20260806 02:50:37.995543 28328 raft_consensus.cc:2229] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:37.995683 28328 raft_consensus.cc:2258] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.995873 28329 tablet_replica.cc:330] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: stopping tablet replica
I20260806 02:50:37.995785 27521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:50:37.996021 28329 raft_consensus.cc:2229] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 1 LEADER]: Raft consensus shutting down.
W20260806 02:50:37.996263 27521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:37.996362 28329 raft_consensus.cc:2258] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.997118 28328 ts_tablet_manager.cc:1905] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:38.000000 28328 ts_tablet_manager.cc:1918] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:38.000093 28328 log.cc:1196] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/327da5a492354381a919fe3b10f5cc6d
I20260806 02:50:38.000413 28328 ts_tablet_manager.cc:1939] T 327da5a492354381a919fe3b10f5cc6d P 9f7b3f23babc4727bcf6452de3d61b98: Deleting consensus metadata
I20260806 02:50:38.001312 27504 catalog_manager.cc:4841] TS 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781): tablet 327da5a492354381a919fe3b10f5cc6d (table test-table [id=9beb5370bb54405e854ac184601b6b0b]) successfully deleted
I20260806 02:50:38.001678 28329 ts_tablet_manager.cc:1905] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:38.001801 27911 tablet_service.cc:1467] Processing CreateTablet for tablet 106c5b65fc454fe4b831f21097dd960c (DEFAULT_TABLE table=test-table [id=dd0b5ecb94e140b68a0a963f1fd483ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:38.001825 27702 tablet_service.cc:1467] Processing CreateTablet for tablet 106c5b65fc454fe4b831f21097dd960c (DEFAULT_TABLE table=test-table [id=dd0b5ecb94e140b68a0a963f1fd483ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:38.002068 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c5b65fc454fe4b831f21097dd960c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:38.002077 27911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c5b65fc454fe4b831f21097dd960c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:38.003852 28102 tablet_service.cc:1467] Processing CreateTablet for tablet 106c5b65fc454fe4b831f21097dd960c (DEFAULT_TABLE table=test-table [id=dd0b5ecb94e140b68a0a963f1fd483ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:38.004148 28102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c5b65fc454fe4b831f21097dd960c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:38.004436 28341 tablet_bootstrap.cc:492] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc: Bootstrap starting.
I20260806 02:50:38.005432 28341 tablet_bootstrap.cc:654] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:38.006081 28101 tablet_service.cc:1514] Processing DeleteTablet for tablet 327da5a492354381a919fe3b10f5cc6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:34908
I20260806 02:50:38.006111 28340 tablet_bootstrap.cc:492] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Bootstrap starting.
I20260806 02:50:38.006269 28327 tablet_replica.cc:330] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: stopping tablet replica
I20260806 02:50:38.006376 28327 raft_consensus.cc:2229] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:38.006489 28327 raft_consensus.cc:2258] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:38.007099 28340 tablet_bootstrap.cc:654] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:38.007301 28327 ts_tablet_manager.cc:1905] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:38.008445 28340 tablet_bootstrap.cc:492] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: No bootstrap required, opened a new log
I20260806 02:50:38.008538 28340 ts_tablet_manager.cc:1397] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:50:38.008980 28340 raft_consensus.cc:348] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.009106 28340 raft_consensus.cc:374] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:38.009140 28340 raft_consensus.cc:729] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7b3f23babc4727bcf6452de3d61b98, State: Initialized, Role: FOLLOWER
I20260806 02:50:38.009274 28340 consensus_queue.cc:260] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.009542 28340 ts_tablet_manager.cc:1428] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:38.009682 28327 ts_tablet_manager.cc:1918] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:38.009776 28327 log.cc:1196] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/327da5a492354381a919fe3b10f5cc6d
I20260806 02:50:38.010030 28327 ts_tablet_manager.cc:1939] T 327da5a492354381a919fe3b10f5cc6d P e0732da08d29462a8b32d999b773f83e: Deleting consensus metadata
I20260806 02:50:38.010099 28341 tablet_bootstrap.cc:492] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc: No bootstrap required, opened a new log
I20260806 02:50:38.010200 28341 ts_tablet_manager.cc:1397] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:50:38.010638 28341 raft_consensus.cc:348] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.010735 27505 catalog_manager.cc:4841] TS e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251): tablet 327da5a492354381a919fe3b10f5cc6d (table test-table [id=9beb5370bb54405e854ac184601b6b0b]) successfully deleted
I20260806 02:50:38.010779 28341 raft_consensus.cc:374] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:38.010811 28341 raft_consensus.cc:729] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cef48f52043ca8c1347e5cc90ebfc, State: Initialized, Role: FOLLOWER
I20260806 02:50:38.010951 28341 consensus_queue.cc:260] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.011055 28342 tablet_bootstrap.cc:492] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Bootstrap starting.
I20260806 02:50:38.011221 28341 ts_tablet_manager.cc:1428] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:50:38.011866 28342 tablet_bootstrap.cc:654] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:38.013677 28329 ts_tablet_manager.cc:1918] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:50:38.013768 28329 log.cc:1196] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/327da5a492354381a919fe3b10f5cc6d
I20260806 02:50:38.014065 28329 ts_tablet_manager.cc:1939] T 327da5a492354381a919fe3b10f5cc6d P 086cef48f52043ca8c1347e5cc90ebfc: Deleting consensus metadata
I20260806 02:50:38.014801 27505 catalog_manager.cc:4841] TS 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397): tablet 327da5a492354381a919fe3b10f5cc6d (table test-table [id=9beb5370bb54405e854ac184601b6b0b]) successfully deleted
I20260806 02:50:38.016199 28342 tablet_bootstrap.cc:492] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: No bootstrap required, opened a new log
I20260806 02:50:38.016296 28342 ts_tablet_manager.cc:1397] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260806 02:50:38.016724 28342 raft_consensus.cc:348] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.016841 28342 raft_consensus.cc:374] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:38.016887 28342 raft_consensus.cc:729] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0732da08d29462a8b32d999b773f83e, State: Initialized, Role: FOLLOWER
I20260806 02:50:38.017016 28342 consensus_queue.cc:260] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.017278 28342 ts_tablet_manager.cc:1428] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:50:38.247802 28351 raft_consensus.cc:482] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:38.248021 28351 raft_consensus.cc:504] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.249234 28351 leader_election.cc:288] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397), e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251)
I20260806 02:50:38.254020 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.250072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:45931 (local address 127.24.130.131:41251)
0806 02:50:38.250227 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.250231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.250425 (+ 194us) server_negotiation.cc:407] Connection header received
0806 02:50:38.250553 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.250559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.250622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.250722 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:38.251026 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.251579 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.252725 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.253330 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.253485 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.253547 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.253808 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.253883 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:50:38.254020 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.249646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:49077 (local address 127.24.130.129:36397)
0806 02:50:38.249790 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.249795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.249922 (+ 127us) server_negotiation.cc:407] Connection header received
0806 02:50:38.250124 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.250128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.250194 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.250279 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:38.250681 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.251333 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.252815 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.253429 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.253540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.253592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.253808 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.253902 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:50:38.254137 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.249490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36397 (local address 127.24.130.130:49077)
0806 02:50:38.249778 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:38.249796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:38.249926 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:38.250310 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:38.250313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:38.250332 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:38.250518 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.250527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.251479 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.251482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.252641 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.252652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.253530 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.253533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.253640 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.253696 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:50:38.253773 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:50:38.253900 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":25,"thread_start_us":92,"threads_started":1}
I20260806 02:50:38.254140 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.249908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.24.130.130:45931)
0806 02:50:38.250316 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:38.250332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:38.250437 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:38.250733 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:38.250736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:38.250755 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:38.250905 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.250912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.251711 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.251713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.252584 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.252593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.253467 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.253469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.253598 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.253690 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 02:50:38.253773 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:50:38.253900 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":121,"thread_start_us":135,"threads_started":1}
I20260806 02:50:38.254686 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c5b65fc454fe4b831f21097dd960c" candidate_uuid: "9f7b3f23babc4727bcf6452de3d61b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc" is_pre_election: true
I20260806 02:50:38.254644 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c5b65fc454fe4b831f21097dd960c" candidate_uuid: "9f7b3f23babc4727bcf6452de3d61b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0732da08d29462a8b32d999b773f83e" is_pre_election: true
I20260806 02:50:38.254873 28129 raft_consensus.cc:2454] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f7b3f23babc4727bcf6452de3d61b98 in term 0.
I20260806 02:50:38.254869 27728 raft_consensus.cc:2454] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f7b3f23babc4727bcf6452de3d61b98 in term 0.
I20260806 02:50:38.255254 27836 leader_election.cc:302] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7b3f23babc4727bcf6452de3d61b98, e0732da08d29462a8b32d999b773f83e; no voters:
I20260806 02:50:38.255548 28351 raft_consensus.cc:2781] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:38.255625 28351 raft_consensus.cc:482] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:38.255681 28351 raft_consensus.cc:3037] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:38.257172 28351 raft_consensus.cc:504] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.257550 28351 leader_election.cc:288] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [CANDIDATE]: Term 1 election: Requested vote from peers 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397), e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251)
I20260806 02:50:38.258005 28129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c5b65fc454fe4b831f21097dd960c" candidate_uuid: "9f7b3f23babc4727bcf6452de3d61b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0732da08d29462a8b32d999b773f83e"
I20260806 02:50:38.257992 27728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c5b65fc454fe4b831f21097dd960c" candidate_uuid: "9f7b3f23babc4727bcf6452de3d61b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cef48f52043ca8c1347e5cc90ebfc"
I20260806 02:50:38.258168 27728 raft_consensus.cc:3037] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:38.258176 28129 raft_consensus.cc:3037] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:38.259572 27728 raft_consensus.cc:2454] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7b3f23babc4727bcf6452de3d61b98 in term 1.
I20260806 02:50:38.259675 28129 raft_consensus.cc:2454] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7b3f23babc4727bcf6452de3d61b98 in term 1.
I20260806 02:50:38.259919 27836 leader_election.cc:302] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cef48f52043ca8c1347e5cc90ebfc, 9f7b3f23babc4727bcf6452de3d61b98; no voters:
I20260806 02:50:38.260146 28351 raft_consensus.cc:2781] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:38.260434 28351 raft_consensus.cc:686] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 LEADER]: Becoming Leader. State: Replica: 9f7b3f23babc4727bcf6452de3d61b98, State: Running, Role: LEADER
I20260806 02:50:38.260766 28351 consensus_queue.cc:237] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } }
I20260806 02:50:38.263949 27521 catalog_manager.cc:5495] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130). New cstate: current_term: 1 leader_uuid: "9f7b3f23babc4727bcf6452de3d61b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7b3f23babc4727bcf6452de3d61b98" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:38.320474 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.316636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.24.130.130:38781)
0806 02:50:38.316787 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.316794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.316810 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:50:38.317031 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.317034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.317090 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.317211 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:38.317468 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.318004 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.318910 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.319138 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.319254 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.319556 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:38.320047 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:38.320185 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.320258 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.320319 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:50:38.331184 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.328026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:48511 (local address 127.24.130.131:41251)
0806 02:50:38.328154 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.328157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.328171 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:38.328219 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.328221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.328265 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.328334 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:38.328631 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.329137 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.330065 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.330365 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.330447 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.330751 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:38.330824 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:38.330889 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.331009 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.331057 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:50:38.331229 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.327686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:41251 (local address 127.24.130.130:48511)
0806 02:50:38.327987 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:38.328002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:38.328088 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:38.328402 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:38.328405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:38.328413 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:38.328525 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.328530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.329278 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.329281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.329931 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.329940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.330490 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.330493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.330568 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.330617 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:38.330904 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:38.330909 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:50:38.330964 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 02:50:38.331032 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260806 02:50:38.337838 28129 raft_consensus.cc:1261] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Refusing update from remote peer 9f7b3f23babc4727bcf6452de3d61b98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:38.337908 27728 raft_consensus.cc:1261] T 106c5b65fc454fe4b831f21097dd960c P 086cef48f52043ca8c1347e5cc90ebfc [term 1 FOLLOWER]: Refusing update from remote peer 9f7b3f23babc4727bcf6452de3d61b98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:38.338352 28351 consensus_queue.cc:1035] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "086cef48f52043ca8c1347e5cc90ebfc" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:38.338562 28382 consensus_queue.cc:1035] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0732da08d29462a8b32d999b773f83e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:38.356397 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.353085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:41475 (local address 127.24.130.130:38781)
0806 02:50:38.353217 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.353222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.353234 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:50:38.353288 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.353291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.353347 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.353445 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:38.353688 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.354235 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.355239 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.355561 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.355688 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.355943 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:38.356036 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:38.356111 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.356233 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.356288 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:50:38.356483 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.352692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:38781 (local address 127.24.130.131:41475)
0806 02:50:38.352852 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:38.352871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:38.352955 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:38.353462 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:38.353465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:38.353473 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:50:38.353575 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.353580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.354362 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.354366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.355117 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:38.355127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.355682 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.355686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:38.355780 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.355820 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:50:38.356138 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:50:38.356149 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:50:38.356225 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:50:38.356339 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:50:38.357247 28297 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:50:39.919564 27521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:39.919736 27521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:39.922376 27521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dc26063ceb042bcb47cd680579ee549: Sending DeleteTablet for 3 replicas of tablet 106c5b65fc454fe4b831f21097dd960c
I20260806 02:50:39.923108 27702 tablet_service.cc:1514] Processing DeleteTablet for tablet 106c5b65fc454fe4b831f21097dd960c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:39 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:37894
I20260806 02:50:39.923175 28102 tablet_service.cc:1514] Processing DeleteTablet for tablet 106c5b65fc454fe4b831f21097dd960c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:39 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:34908
I20260806 02:50:39.923244 27911 tablet_service.cc:1514] Processing DeleteTablet for tablet 106c5b65fc454fe4b831f21097dd960c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:39 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:56152
I20260806 02:50:39.923533 28424 tablet_replica.cc:330] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: stopping tablet replica
I20260806 02:50:39.923635 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 27620
I20260806 02:50:39.935792 28426 tablet_replica.cc:330] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: stopping tablet replica
I20260806 02:50:39.936007 28426 raft_consensus.cc:2229] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:39.936458 28426 raft_consensus.cc:2258] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:39.923691 28424 raft_consensus.cc:2229] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:39.936879 28424 raft_consensus.cc:2258] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:39.937650 28424 ts_tablet_manager.cc:1905] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:39.937650 28426 ts_tablet_manager.cc:1905] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:39.940693 28424 ts_tablet_manager.cc:1918] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:50:39.940702 28426 ts_tablet_manager.cc:1918] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:50:39.940794 28424 log.cc:1196] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/106c5b65fc454fe4b831f21097dd960c
I20260806 02:50:39.940794 28426 log.cc:1196] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/106c5b65fc454fe4b831f21097dd960c
I20260806 02:50:39.941095 28424 ts_tablet_manager.cc:1939] T 106c5b65fc454fe4b831f21097dd960c P e0732da08d29462a8b32d999b773f83e: Deleting consensus metadata
I20260806 02:50:39.941102 28426 ts_tablet_manager.cc:1939] T 106c5b65fc454fe4b831f21097dd960c P 9f7b3f23babc4727bcf6452de3d61b98: Deleting consensus metadata
I20260806 02:50:39.941996 27504 catalog_manager.cc:4841] TS 9f7b3f23babc4727bcf6452de3d61b98 (127.24.130.130:38781): tablet 106c5b65fc454fe4b831f21097dd960c (table test-table [id=dd0b5ecb94e140b68a0a963f1fd483ff]) successfully deleted
I20260806 02:50:39.942255 27505 catalog_manager.cc:4841] TS e0732da08d29462a8b32d999b773f83e (127.24.130.131:41251): tablet 106c5b65fc454fe4b831f21097dd960c (table test-table [id=dd0b5ecb94e140b68a0a963f1fd483ff]) successfully deleted
W20260806 02:50:39.945605 27505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:36397 (error 108)
I20260806 02:50:39.945940 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 27816
I20260806 02:50:39.952772 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:39.950640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36397 (local address 127.0.0.1:38102)
0806 02:50:39.952433 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:39.952580 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1600,"thread_start_us":1442,"threads_started":1}
W20260806 02:50:39.952982 27505 catalog_manager.cc:4584] TS 086cef48f52043ca8c1347e5cc90ebfc (127.24.130.129:36397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 106c5b65fc454fe4b831f21097dd960c: Network error: Client connection negotiation failed: client connection to 127.24.130.129:36397: connect: Connection refused (error 111)
I20260806 02:50:39.961213 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 28014
I20260806 02:50:39.976142 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 27473
2026-08-06T02:50:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7133 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:40 dist-test-slave-3kk6 krb5kdc[28433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:40 dist-test-slave-3kk6 krb5kdc[28433](info): set up 2 sockets
Aug 06 02:50:40 dist-test-slave-3kk6 krb5kdc[28433](info): commencing operation
krb5kdc: starting...
W20260806 02:50:41.251219 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:41 dist-test-slave-3kk6 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:41.389468 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44415
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:41.508766 28449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:41.509150 28449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:41.509196 28449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:41.513589 28449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:41.513681 28449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:41.513701 28449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:41.513718 28449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:41.513734 28449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:41.519438 28449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:41.520524 28449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:41.521543 28449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:41.530315 28457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:41.530315 28456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:41.530303 28459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:41.530565 28449 server_base.cc:1034] running on GCE node
I20260806 02:50:41.531356 28449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:41.532519 28449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:41.533744 28449 hybrid_clock.cc:648] HybridClock initialized: now 1785984641533716 us; error 47 us; skew 500 ppm
Aug 06 02:50:41 dist-test-slave-3kk6 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:41.536959 28449 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:41.537997 28449 webserver.cc:460] Webserver started at http://127.24.130.190:36125/ using document root <none> and password file <none>
I20260806 02:50:41.538586 28449 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:41.538725 28449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:41.538985 28449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:41.541162 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "1ca921ef05f244a8828f3631d2992e3f"
format_stamp: "Formatted at 2026-08-06 02:50:41 on dist-test-slave-3kk6"
server_key: "53e98f959514309e9ff3a7b4f7ef0b7e"
server_key_iv: "6a2f857f1e6594c321102eeba44415f5"
server_key_version: "encryptionkey@0"
I20260806 02:50:41.541800 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "1ca921ef05f244a8828f3631d2992e3f"
format_stamp: "Formatted at 2026-08-06 02:50:41 on dist-test-slave-3kk6"
server_key: "53e98f959514309e9ff3a7b4f7ef0b7e"
server_key_iv: "6a2f857f1e6594c321102eeba44415f5"
server_key_version: "encryptionkey@0"
I20260806 02:50:41.545830 28449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:50:41.548833 28468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:41.550235 28449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:41.550413 28449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "1ca921ef05f244a8828f3631d2992e3f"
format_stamp: "Formatted at 2026-08-06 02:50:41 on dist-test-slave-3kk6"
server_key: "53e98f959514309e9ff3a7b4f7ef0b7e"
server_key_iv: "6a2f857f1e6594c321102eeba44415f5"
server_key_version: "encryptionkey@0"
I20260806 02:50:41.550554 28449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:41.564790 28449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:41.568521 28449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:41.568754 28449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:41.576591 28449 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:44415
I20260806 02:50:41.576608 28538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:44415 every 8 connection(s)
I20260806 02:50:41.577704 28449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:41.580753 28539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:41.586540 28539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f: Bootstrap starting.
I20260806 02:50:41.587872 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 28449
I20260806 02:50:41.588003 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:41.588272 25098 external_mini_cluster.cc:1428] Setting key 79c3a5bfbf3e1ab4b5d98d9eddc52154
I20260806 02:50:41.589370 28539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:41.590227 28539 log.cc:824] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:41.592651 28539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f: No bootstrap required, opened a new log
I20260806 02:50:41.597177 28539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca921ef05f244a8828f3631d2992e3f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44415 } }
I20260806 02:50:41.597524 28539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:41.597615 28539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca921ef05f244a8828f3631d2992e3f, State: Initialized, Role: FOLLOWER
Aug 06 02:50:41 dist-test-slave-3kk6 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:41.598099 28539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca921ef05f244a8828f3631d2992e3f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44415 } }
I20260806 02:50:41.598271 28539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:41.598364 28539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:41.598469 28539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:41.599555 28539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca921ef05f244a8828f3631d2992e3f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44415 } }
I20260806 02:50:41.599936 28539 leader_election.cc:302] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ca921ef05f244a8828f3631d2992e3f; no voters:
I20260806 02:50:41.600319 28539 leader_election.cc:288] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:41.600409 28546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:41.600596 28546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [term 1 LEADER]: Becoming Leader. State: Replica: 1ca921ef05f244a8828f3631d2992e3f, State: Running, Role: LEADER
I20260806 02:50:41.600921 28546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca921ef05f244a8828f3631d2992e3f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44415 } }
I20260806 02:50:41.601694 28539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:41.602753 28548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ca921ef05f244a8828f3631d2992e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca921ef05f244a8828f3631d2992e3f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44415 } } }
I20260806 02:50:41.602888 28548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:41.603180 28547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca921ef05f244a8828f3631d2992e3f. Latest consensus state: current_term: 1 leader_uuid: "1ca921ef05f244a8828f3631d2992e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca921ef05f244a8828f3631d2992e3f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44415 } } }
I20260806 02:50:41.603288 28547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:41.603307 28563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:41.605228 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.589548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.24.130.190:44415)
0806 02:50:41.589992 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:41.590009 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:41.590046 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:50:41.590928 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:41.590955 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:41.591340 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:41.591652 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:41.592629 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.593654 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.594846 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.595326 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.595456 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:41.598753 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:41.598759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:41.598774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:41.598803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:41.600938 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:41.601498 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:41.601504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:41.601510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:41.601638 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:41.601954 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:41.601960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:41.601965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:41.602571 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:41.603744 (+ 1173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:41.603915 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 02:50:41.604238 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20260806 02:50:41.606796 28563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:41.612083 28563 catalog_manager.cc:1347] Generated new cluster ID: 201957043f884f938852eb9620fb82d5
I20260806 02:50:41.612157 28563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:41.617583 28563 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:41.618845 28563 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:41.625226 28563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ca921ef05f244a8828f3631d2992e3f: Generated new TSK 0
I20260806 02:50:41.626089 28563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:41.665951 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44415
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:41.795420 28576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:41.795744 28576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:41.795835 28576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:41.799944 28576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:41.800040 28576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:41.800125 28576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:41.805817 28576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:41.807060 28576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:41.808171 28576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:41.818609 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:41.818588 28576 server_base.cc:1034] running on GCE node
W20260806 02:50:41.818552 28585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:41.818553 28588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:41.819268 28576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:41.819989 28576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:41.821188 28576 hybrid_clock.cc:648] HybridClock initialized: now 1785984641821156 us; error 48 us; skew 500 ppm
Aug 06 02:50:41 dist-test-slave-3kk6 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:41.824186 28576 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:41.825151 28576 webserver.cc:460] Webserver started at http://127.24.130.129:46003/ using document root <none> and password file <none>
I20260806 02:50:41.825742 28576 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:41.825838 28576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:41.826084 28576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:41.828341 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "685919810edf444da422ab598ef47685"
format_stamp: "Formatted at 2026-08-06 02:50:41 on dist-test-slave-3kk6"
server_key: "e6dd7729f20a36ae8fd476ff451887c2"
server_key_iv: "24681c63033b62e0dbc8215b231785f5"
server_key_version: "encryptionkey@0"
I20260806 02:50:41.828961 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "685919810edf444da422ab598ef47685"
format_stamp: "Formatted at 2026-08-06 02:50:41 on dist-test-slave-3kk6"
server_key: "e6dd7729f20a36ae8fd476ff451887c2"
server_key_iv: "24681c63033b62e0dbc8215b231785f5"
server_key_version: "encryptionkey@0"
I20260806 02:50:41.832957 28576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:41.835683 28597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:41.837013 28576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:41.837157 28576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "685919810edf444da422ab598ef47685"
format_stamp: "Formatted at 2026-08-06 02:50:41 on dist-test-slave-3kk6"
server_key: "e6dd7729f20a36ae8fd476ff451887c2"
server_key_iv: "24681c63033b62e0dbc8215b231785f5"
server_key_version: "encryptionkey@0"
I20260806 02:50:41.837283 28576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:41.850615 28576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:41.853832 28576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:41.854063 28576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:41.854849 28576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:41.855813 28576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:41.855887 28576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:41.855960 28576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:41.856009 28576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:41.865806 28576 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:36117
I20260806 02:50:41.865828 28745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:36117 every 8 connection(s)
I20260806 02:50:41.866873 28576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:41.875100 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 28576
I20260806 02:50:41.875236 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:41.875578 25098 external_mini_cluster.cc:1428] Setting key ccf75d03d8201c84a5fe5cd56f32ade8
Aug 06 02:50:41 dist-test-slave-3kk6 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984641, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:41.882596 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.868849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:43145 (local address 127.24.130.190:44415)
0806 02:50:41.869003 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:41.869007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:41.869904 (+ 897us) server_negotiation.cc:407] Connection header received
0806 02:50:41.871203 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:41.871206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:41.871259 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:41.871350 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:41.872728 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.873263 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.874418 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.874646 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.874798 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:41.877873 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:41.877880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:41.877885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:41.877921 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:41.879812 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:41.880612 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:41.880615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:41.880616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:41.880713 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:41.881210 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:41.881213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:41.881215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:41.881465 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:41.881565 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:41.882320 (+ 755us) server_negotiation.cc:299] Negotiation successful
0806 02:50:41.882461 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:41.884228 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.869241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44415 (local address 127.24.130.129:43145)
0806 02:50:41.869717 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.869762 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:41.870976 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:41.871513 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:41.871520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:41.871925 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:41.872413 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:41.872426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.873416 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.873419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:41.874296 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:41.874306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.874893 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.874900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:41.875123 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:41.875879 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:41.875902 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:41.877682 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:41.879969 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:41.879980 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:41.880000 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:41.880476 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:41.880828 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:41.880834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:41.880837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:41.881070 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:41.881599 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:41.881609 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:41.881996 (+ 387us) client_negotiation.cc:769] Sending connection context
0806 02:50:41.882236 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 02:50:41.882717 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260806 02:50:41.885931 28746 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44415
I20260806 02:50:41.886241 28746 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:41.886889 28746 heartbeater.cc:511] Master 127.24.130.190:44415 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:50:41.888593 28494 ts_manager.cc:194] Registered new tserver with Master: 685919810edf444da422ab598ef47685 (127.24.130.129:36117)
I20260806 02:50:41.890144 28494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:43145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:41.909790 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44415
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:42.029464 28752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:42.029788 28752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:42.029865 28752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:42.033982 28752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:42.034070 28752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:42.034155 28752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:42.039645 28752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:42.040896 28752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:42.042069 28752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:42.052179 28758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:42.052270 28761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:42.052341 28752 server_base.cc:1034] running on GCE node
W20260806 02:50:42.052211 28759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:42.052877 28752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:42.053531 28752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:42.054762 28752 hybrid_clock.cc:648] HybridClock initialized: now 1785984642054727 us; error 42 us; skew 500 ppm
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:42.057811 28752 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:42.058809 28752 webserver.cc:460] Webserver started at http://127.24.130.130:40993/ using document root <none> and password file <none>
I20260806 02:50:42.059355 28752 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:42.059418 28752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:42.059594 28752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:42.061679 28752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "c557e14964ae43f1b1861443169dec6a"
format_stamp: "Formatted at 2026-08-06 02:50:42 on dist-test-slave-3kk6"
server_key: "ae262d52546a12232d2bf835da7c21ba"
server_key_iv: "7edcb024a25f98f0931c5b1fbad17d2b"
server_key_version: "encryptionkey@0"
I20260806 02:50:42.062227 28752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "c557e14964ae43f1b1861443169dec6a"
format_stamp: "Formatted at 2026-08-06 02:50:42 on dist-test-slave-3kk6"
server_key: "ae262d52546a12232d2bf835da7c21ba"
server_key_iv: "7edcb024a25f98f0931c5b1fbad17d2b"
server_key_version: "encryptionkey@0"
I20260806 02:50:42.066140 28752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 02:50:42.068657 28768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:42.069881 28752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:42.069980 28752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "c557e14964ae43f1b1861443169dec6a"
format_stamp: "Formatted at 2026-08-06 02:50:42 on dist-test-slave-3kk6"
server_key: "ae262d52546a12232d2bf835da7c21ba"
server_key_iv: "7edcb024a25f98f0931c5b1fbad17d2b"
server_key_version: "encryptionkey@0"
I20260806 02:50:42.070119 28752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:42.093324 28752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:42.097527 28752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:42.097836 28752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:42.098573 28752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:42.099651 28752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:42.099704 28752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:42.099766 28752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:42.099787 28752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:42.113662 28752 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:35741
I20260806 02:50:42.113822 28919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:35741 every 8 connection(s)
I20260806 02:50:42.114743 28752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:42.122244 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 28752
I20260806 02:50:42.122341 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:42.122705 25098 external_mini_cluster.cc:1428] Setting key 840c07787e4038090701d21ff0560b90
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:42.129652 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:42.116690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:53277 (local address 127.24.130.190:44415)
0806 02:50:42.116862 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:42.116872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:42.117642 (+ 770us) server_negotiation.cc:407] Connection header received
0806 02:50:42.119058 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:42.119061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:42.119124 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:42.119198 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:42.120373 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.120922 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.122157 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.122450 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.122541 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:42.125481 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:42.125490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:42.125492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:42.125517 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:42.127397 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:42.127915 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:42.127918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:42.127919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:42.127999 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:42.128320 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:42.128326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:42.128331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:42.128634 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:42.128749 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:42.129229 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:50:42.129390 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:50:42.130260 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:42.117013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44415 (local address 127.24.130.130:53277)
0806 02:50:42.117480 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:42.117525 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:42.118831 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:42.119371 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:42.119377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:42.119746 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:42.120222 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:42.120237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.121073 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.121076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:42.122013 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:42.122028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.122617 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.122622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:42.122900 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:42.123621 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:42.123641 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:42.125302 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:42.127535 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:42.127539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:42.127551 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:42.127803 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:42.128083 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:42.128085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:42.128087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:42.128225 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:42.128743 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:42.128748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:42.128978 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 02:50:42.129136 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:50:42.129399 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260806 02:50:42.131525 28922 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44415
I20260806 02:50:42.131765 28922 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:42.132262 28922 heartbeater.cc:511] Master 127.24.130.190:44415 requested a full tablet report, sending...
I20260806 02:50:42.133356 28494 ts_manager.cc:194] Registered new tserver with Master: c557e14964ae43f1b1861443169dec6a (127.24.130.130:35741)
I20260806 02:50:42.133970 28494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:53277
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:42.154798 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44415
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:42.274686 28927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:42.274996 28927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:42.275041 28927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:42.279115 28927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:42.279201 28927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:42.279263 28927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:42.284825 28927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:42.286156 28927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:42.287214 28927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:42.297500 28936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:42.297501 28939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:42.297642 28927 server_base.cc:1034] running on GCE node
W20260806 02:50:42.297508 28937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:42.298203 28927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:42.298970 28927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:42.300161 28927 hybrid_clock.cc:648] HybridClock initialized: now 1785984642300140 us; error 34 us; skew 500 ppm
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[28433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:42.303284 28927 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:42.304296 28927 webserver.cc:460] Webserver started at http://127.24.130.131:37171/ using document root <none> and password file <none>
I20260806 02:50:42.304836 28927 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:42.304895 28927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:42.305068 28927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:42.307262 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "381437bbbb4a4e33a1d02a4372ae1a93"
format_stamp: "Formatted at 2026-08-06 02:50:42 on dist-test-slave-3kk6"
server_key: "f7a5843de13535cfa6a800d6590c9fc6"
server_key_iv: "2faa9acbcc928b843e99ec1f43c5e821"
server_key_version: "encryptionkey@0"
I20260806 02:50:42.307838 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "381437bbbb4a4e33a1d02a4372ae1a93"
format_stamp: "Formatted at 2026-08-06 02:50:42 on dist-test-slave-3kk6"
server_key: "f7a5843de13535cfa6a800d6590c9fc6"
server_key_iv: "2faa9acbcc928b843e99ec1f43c5e821"
server_key_version: "encryptionkey@0"
I20260806 02:50:42.311810 28927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:50:42.314287 28952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:42.315721 28927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:42.315843 28927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "381437bbbb4a4e33a1d02a4372ae1a93"
format_stamp: "Formatted at 2026-08-06 02:50:42 on dist-test-slave-3kk6"
server_key: "f7a5843de13535cfa6a800d6590c9fc6"
server_key_iv: "2faa9acbcc928b843e99ec1f43c5e821"
server_key_version: "encryptionkey@0"
I20260806 02:50:42.315943 28927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:42.339747 28927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:42.343286 28927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:42.343469 28927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:42.344099 28927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:42.344980 28927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:42.345029 28927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:42.345073 28927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:42.345089 28927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:42.354851 28927 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:41617
I20260806 02:50:42.354923 29111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:41617 every 8 connection(s)
I20260806 02:50:42.355890 28927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:42.363337 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 28927
I20260806 02:50:42.363452 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:42.363762 25098 external_mini_cluster.cc:1428] Setting key dd8fae17cb1f1fe58c822afc7326b5ec
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[28433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:42.370352 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:42.357915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:57117 (local address 127.24.130.190:44415)
0806 02:50:42.358085 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:42.358090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:42.359099 (+ 1009us) server_negotiation.cc:407] Connection header received
0806 02:50:42.360428 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:42.360431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:42.360483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:42.360565 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:42.361807 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.362371 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.363631 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.363907 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.364043 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:42.367101 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:42.367105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:42.367107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:42.367141 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:42.368508 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:42.368994 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:42.368997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:42.368998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:42.369078 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:42.369399 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:42.369401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:42.369402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:42.369573 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:42.369647 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:42.370130 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 02:50:42.370228 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:50:42.371253 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:42.358380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44415 (local address 127.24.130.131:57117)
0806 02:50:42.358921 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:42.358970 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:42.360197 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:42.360743 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:42.360750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:42.361154 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:42.361645 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:42.361659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.362487 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.362489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:42.363434 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:42.363448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:42.364087 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:42.364092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:42.364315 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:42.365070 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:42.365090 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:42.366937 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:42.368616 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:42.368621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:42.368633 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:42.368896 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:42.369160 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:42.369162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:42.369163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:42.369306 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:42.369662 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:42.369667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:42.369916 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 02:50:42.370066 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:50:42.370319 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":126,"threads_started":1}
I20260806 02:50:42.372375 29114 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44415
I20260806 02:50:42.372632 29114 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:42.373188 29114 heartbeater.cc:511] Master 127.24.130.190:44415 requested a full tablet report, sending...
I20260806 02:50:42.374212 28494 ts_manager.cc:194] Registered new tserver with Master: 381437bbbb4a4e33a1d02a4372ae1a93 (127.24.130.131:41617)
I20260806 02:50:42.374923 28494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:57117
I20260806 02:50:42.377758 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:50:42.393010 28543 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:42.391316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.24.130.190:44415)
0806 02:50:42.391483 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:42.391490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:42.391667 (+ 177us) server_negotiation.cc:407] Connection header received
0806 02:50:42.391818 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:42.391821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:42.391868 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:42.391944 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:42.392838 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35594: BlockingRecv error: recv got EOF from 127.0.0.1:35594 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:50:42.394140 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 28576
I20260806 02:50:42.409655 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 28752
I20260806 02:50:42.422331 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 28927
I20260806 02:50:42.431531 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 28449
2026-08-06T02:50:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2446 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[29128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[29128](info): set up 2 sockets
Aug 06 02:50:42 dist-test-slave-3kk6 krb5kdc[29128](info): commencing operation
krb5kdc: starting...
W20260806 02:50:43.646744 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:43 dist-test-slave-3kk6 krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:43.782074 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43193
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:43.897557 29151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:43.897858 29151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:43.897902 29151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:43.902056 29151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:43.902158 29151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:43.902179 29151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:43.902195 29151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:43.902211 29151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:43.907796 29151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:43.908921 29151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:43.909946 29151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:43.918776 29161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:43.918994 29151 server_base.cc:1034] running on GCE node
W20260806 02:50:43.918774 29159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:43.918771 29158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:43.919713 29151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:43.920863 29151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:43.922029 29151 hybrid_clock.cc:648] HybridClock initialized: now 1785984643922001 us; error 41 us; skew 500 ppm
Aug 06 02:50:43 dist-test-slave-3kk6 krb5kdc[29128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:43.924861 29151 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:43.925803 29151 webserver.cc:460] Webserver started at http://127.24.130.190:38997/ using document root <none> and password file <none>
I20260806 02:50:43.926369 29151 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:43.926465 29151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:43.926769 29151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:43.928930 29151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66"
format_stamp: "Formatted at 2026-08-06 02:50:43 on dist-test-slave-3kk6"
server_key: "00990e2bc6d408ca7ff7f82d0649d207"
server_key_iv: "cb3c2807eaa5dcad3db357b9310404df"
server_key_version: "encryptionkey@0"
I20260806 02:50:43.929546 29151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66"
format_stamp: "Formatted at 2026-08-06 02:50:43 on dist-test-slave-3kk6"
server_key: "00990e2bc6d408ca7ff7f82d0649d207"
server_key_iv: "cb3c2807eaa5dcad3db357b9310404df"
server_key_version: "encryptionkey@0"
I20260806 02:50:43.933440 29151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:43.935962 29172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:43.937263 29151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:43.937414 29151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66"
format_stamp: "Formatted at 2026-08-06 02:50:43 on dist-test-slave-3kk6"
server_key: "00990e2bc6d408ca7ff7f82d0649d207"
server_key_iv: "cb3c2807eaa5dcad3db357b9310404df"
server_key_version: "encryptionkey@0"
I20260806 02:50:43.937539 29151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:43.960839 29151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:43.964254 29151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:43.964493 29151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:43.972366 29151 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43193
I20260806 02:50:43.972677 29245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43193 every 8 connection(s)
I20260806 02:50:43.973429 29151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:43.976406 29246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:43.980010 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 29151
I20260806 02:50:43.980166 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:43.980465 25098 external_mini_cluster.cc:1428] Setting key 2ab32401ecfe22e055ddd2072c63f82d
I20260806 02:50:43.982463 29246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66: Bootstrap starting.
I20260806 02:50:43.985074 29246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:43.985837 29246 log.cc:824] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:43.988087 29246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66: No bootstrap required, opened a new log
Aug 06 02:50:43 dist-test-slave-3kk6 krb5kdc[29128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:43.992166 29246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43193 } }
I20260806 02:50:43.992409 29246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:43.992496 29246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec84fa6b6c84d2e9b8c82c474d59e66, State: Initialized, Role: FOLLOWER
I20260806 02:50:43.992991 29246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43193 } }
I20260806 02:50:43.993131 29246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:43.993221 29246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:43.993350 29246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:43.994688 29246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43193 } }
I20260806 02:50:43.995110 29246 leader_election.cc:302] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec84fa6b6c84d2e9b8c82c474d59e66; no voters:
I20260806 02:50:43.995550 29246 leader_election.cc:288] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:43.995683 29253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:43.995999 29253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [term 1 LEADER]: Becoming Leader. State: Replica: 2ec84fa6b6c84d2e9b8c82c474d59e66, State: Running, Role: LEADER
I20260806 02:50:43.996419 29253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43193 } }
I20260806 02:50:43.997032 29246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:43.997581 29251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:43.981980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.24.130.190:43193)
0806 02:50:43.982450 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:43.982467 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:43.982504 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:50:43.983278 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:43.983296 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:43.983627 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:43.983917 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:43.984921 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:43.986074 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:43.987166 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:43.987635 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:43.987773 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:43.990920 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:43.990926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:43.990941 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:43.990971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:43.993674 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:43.994230 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:43.994235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:43.994240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:43.994363 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:43.994750 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:43.994754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:43.994756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:43.995315 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:43.995478 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:43.996230 (+ 752us) server_negotiation.cc:299] Negotiation successful
0806 02:50:43.996563 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":179,"threads_started":1}
I20260806 02:50:43.997659 29253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec84fa6b6c84d2e9b8c82c474d59e66. Latest consensus state: current_term: 1 leader_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43193 } } }
I20260806 02:50:43.997923 29253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:43.998250 29255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec84fa6b6c84d2e9b8c82c474d59e66" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43193 } } }
I20260806 02:50:43.998355 29255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:43.998534 29268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:44.001493 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:44.001878 29268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:44.008595 29268 catalog_manager.cc:1347] Generated new cluster ID: 353b804d882a443dbf8e5d6d3116b620
I20260806 02:50:44.008740 29268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:44.021957 29268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:44.023236 29268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:44.032101 29251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:44.025294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.24.130.190:43193)
0806 02:50:44.025557 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:44.025563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:44.025826 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:50:44.025972 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:44.025976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:44.026027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:44.026133 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:44.026790 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:44.027685 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:44.029267 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:44.029686 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:44.029788 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:44.030876 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:44.030880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:44.030883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:44.031536 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:44.031574 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:44.031595 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:44.031720 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:44.031846 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:50:44.031931 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260806 02:50:44.035305 29268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ec84fa6b6c84d2e9b8c82c474d59e66: Generated new TSK 0
I20260806 02:50:44.036170 29268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:44.045948 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 29151
2026-08-06T02:50:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1618 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:44 dist-test-slave-3kk6 krb5kdc[29292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:44 dist-test-slave-3kk6 krb5kdc[29292](info): set up 2 sockets
Aug 06 02:50:44 dist-test-slave-3kk6 krb5kdc[29292](info): commencing operation
krb5kdc: starting...
W20260806 02:50:45.290339 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:45 dist-test-slave-3kk6 krb5kdc[29292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:45.429701 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36799
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:45.551447 29315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:45.551806 29315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:45.551852 29315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:45.556142 29315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:45.556236 29315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:45.556255 29315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:45.556273 29315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:45.556289 29315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:45.562150 29315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:45.563453 29315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:45.564558 29315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:45.574076 29320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:45.574079 29321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:45.574122 29323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:45.574476 29315 server_base.cc:1034] running on GCE node
I20260806 02:50:45.575150 29315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:45.576403 29315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:45.577580 29315 hybrid_clock.cc:648] HybridClock initialized: now 1785984645577558 us; error 38 us; skew 500 ppm
Aug 06 02:50:45 dist-test-slave-3kk6 krb5kdc[29292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:45.581293 29315 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:45.582410 29315 webserver.cc:460] Webserver started at http://127.24.130.190:38259/ using document root <none> and password file <none>
I20260806 02:50:45.583112 29315 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:45.583186 29315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:45.583436 29315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:45.585753 29315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "d5f361d991154e61ae5a4c0c586a3270"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-3kk6"
server_key: "a14960f2b110bd838a688a6cda1f1223"
server_key_iv: "fe4c5d97906783e5764cbc782cf9da65"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.586406 29315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "d5f361d991154e61ae5a4c0c586a3270"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-3kk6"
server_key: "a14960f2b110bd838a688a6cda1f1223"
server_key_iv: "fe4c5d97906783e5764cbc782cf9da65"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.590878 29315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:50:45.593906 29332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.595542 29315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:45.595729 29315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "d5f361d991154e61ae5a4c0c586a3270"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-3kk6"
server_key: "a14960f2b110bd838a688a6cda1f1223"
server_key_iv: "fe4c5d97906783e5764cbc782cf9da65"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.595891 29315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:45.618487 29315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:45.621826 29315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:45.622069 29315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:45.629930 29315 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:36799
I20260806 02:50:45.630309 29404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:36799 every 8 connection(s)
I20260806 02:50:45.631065 29315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:45.634032 29407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:45.638779 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 29315
I20260806 02:50:45.638926 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:45.639230 25098 external_mini_cluster.cc:1428] Setting key 8b634ad89b3a97a9a042a046f0353809
I20260806 02:50:45.640647 29407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270: Bootstrap starting.
I20260806 02:50:45.643193 29407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:45.644086 29407 log.cc:824] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:45.646472 29407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270: No bootstrap required, opened a new log
Aug 06 02:50:45 dist-test-slave-3kk6 krb5kdc[29292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:45.651811 29407 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f361d991154e61ae5a4c0c586a3270" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36799 } }
I20260806 02:50:45.652158 29407 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:45.652231 29407 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f361d991154e61ae5a4c0c586a3270, State: Initialized, Role: FOLLOWER
I20260806 02:50:45.652842 29407 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f361d991154e61ae5a4c0c586a3270" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36799 } }
I20260806 02:50:45.653041 29407 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:45.653123 29407 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:45.653245 29407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:45.654721 29407 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f361d991154e61ae5a4c0c586a3270" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36799 } }
I20260806 02:50:45.655133 29407 leader_election.cc:302] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f361d991154e61ae5a4c0c586a3270; no voters:
I20260806 02:50:45.655532 29407 leader_election.cc:288] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:45.655721 29412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:45.655975 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.640815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.24.130.190:36799)
0806 02:50:45.641267 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.641284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.641322 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:50:45.641997 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.642015 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.642342 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.642643 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:45.643746 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.644865 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.646072 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.646601 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.646813 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.649833 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:45.649842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:45.649867 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:45.649919 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:45.652210 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.652747 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.652753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.652760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.652914 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.653293 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.653298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.653302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.653956 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:45.654195 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.654421 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.654832 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":152,"threads_started":1}
I20260806 02:50:45.655983 29412 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [term 1 LEADER]: Becoming Leader. State: Replica: d5f361d991154e61ae5a4c0c586a3270, State: Running, Role: LEADER
I20260806 02:50:45.656596 29412 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f361d991154e61ae5a4c0c586a3270" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36799 } }
I20260806 02:50:45.656991 29407 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:45.658846 29413 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f361d991154e61ae5a4c0c586a3270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f361d991154e61ae5a4c0c586a3270" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36799 } } }
I20260806 02:50:45.658996 29413 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:45.658586 29416 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f361d991154e61ae5a4c0c586a3270. Latest consensus state: current_term: 1 leader_uuid: "d5f361d991154e61ae5a4c0c586a3270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f361d991154e61ae5a4c0c586a3270" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36799 } } }
I20260806 02:50:45.659355 29416 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:45.659435 29428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:45.662567 29428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:45.669159 29428 catalog_manager.cc:1347] Generated new cluster ID: 041d193267b746249b8a7ff979f307df
I20260806 02:50:45.669274 29428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:45.679914 29428 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:45.680995 29428 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:45.687114 29428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f361d991154e61ae5a4c0c586a3270: Generated new TSK 0
I20260806 02:50:45.688050 29428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:45.729940 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36799
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:45.857995 29442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:45.858381 29442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:45.858502 29442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:45.862933 29442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:45.863072 29442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:45.863185 29442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:45.869055 29442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:45.870332 29442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:45.871526 29442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:45.882207 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:45.882231 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:45.882207 29453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:45.882630 29442 server_base.cc:1034] running on GCE node
I20260806 02:50:45.883154 29442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:45.883898 29442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:45.885113 29442 hybrid_clock.cc:648] HybridClock initialized: now 1785984645885096 us; error 33 us; skew 500 ppm
Aug 06 02:50:45 dist-test-slave-3kk6 krb5kdc[29292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:45.888417 29442 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:45.889467 29442 webserver.cc:460] Webserver started at http://127.24.130.129:39623/ using document root <none> and password file <none>
I20260806 02:50:45.890125 29442 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:45.890237 29442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:45.890494 29442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:45.892856 29442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "3b9c3aec61d94be9a889db66b1099938"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-3kk6"
server_key: "edc633ba9b17a60ac009c3962c8227cd"
server_key_iv: "6333b96227c30e618fe45694a44dddba"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.893527 29442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "3b9c3aec61d94be9a889db66b1099938"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-3kk6"
server_key: "edc633ba9b17a60ac009c3962c8227cd"
server_key_iv: "6333b96227c30e618fe45694a44dddba"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.898023 29442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 02:50:45.901125 29462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.902480 29442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:45.902638 29442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "3b9c3aec61d94be9a889db66b1099938"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-3kk6"
server_key: "edc633ba9b17a60ac009c3962c8227cd"
server_key_iv: "6333b96227c30e618fe45694a44dddba"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.902861 29442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:45.922328 29442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:45.925909 29442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:45.926153 29442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:45.926915 29442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:45.927835 29442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:45.927912 29442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.927989 29442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:45.928035 29442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.937665 29442 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:44901
I20260806 02:50:45.937693 29631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:44901 every 8 connection(s)
I20260806 02:50:45.938741 29442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:45.939364 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 29442
I20260806 02:50:45.939517 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:45.939857 25098 external_mini_cluster.cc:1428] Setting key c7ec1990b13d8c20ea23e9bc06a80de7
Aug 06 02:50:45 dist-test-slave-3kk6 krb5kdc[29292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:45.954365 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.940745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:52683 (local address 127.24.130.190:36799)
0806 02:50:45.940909 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.940914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.942009 (+ 1095us) server_negotiation.cc:407] Connection header received
0806 02:50:45.943556 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.943559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.943620 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.943745 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:45.945015 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.945564 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.946819 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.947054 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.947168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.950247 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:45.950251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:45.950253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:45.950280 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:45.951929 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.952579 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.952586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.952591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.952686 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.953116 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.953120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.953121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.953381 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:45.953498 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.954049 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.954188 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:50:45.955314 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.941284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36799 (local address 127.24.130.129:52683)
0806 02:50:45.941820 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.941878 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.943322 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.943934 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.943941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.944340 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:45.944851 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.944865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.945706 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.945710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.946620 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.946630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.947243 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.947248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.947421 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.948155 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:45.948173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:45.950035 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:45.952073 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:45.952079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:45.952098 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:45.952391 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:45.952807 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:45.952810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:45.952812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:45.952997 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:45.953498 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:45.953506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:45.953785 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.953991 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.954362 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":163,"threads_started":1}
I20260806 02:50:45.956791 29632 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36799
I20260806 02:50:45.957062 29632 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:45.957595 29632 heartbeater.cc:511] Master 127.24.130.190:36799 requested a full tablet report, sending...
I20260806 02:50:45.959583 29356 ts_manager.cc:194] Registered new tserver with Master: 3b9c3aec61d94be9a889db66b1099938 (127.24.130.129:44901)
I20260806 02:50:45.961377 29356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:52683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:45.974284 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36799
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:46.094612 29640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:46.095091 29640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:46.095192 29640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:46.102931 29640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:46.103052 29640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:46.103156 29640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:46.109793 29640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:46.111037 29640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:46.112165 29640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:46.122138 29647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:46.122144 29649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:46.122417 29640 server_base.cc:1034] running on GCE node
W20260806 02:50:46.122301 29646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:46.122984 29640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:46.123688 29640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:46.124874 29640 hybrid_clock.cc:648] HybridClock initialized: now 1785984646124861 us; error 31 us; skew 500 ppm
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[29292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:46.128017 29640 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:46.128973 29640 webserver.cc:460] Webserver started at http://127.24.130.130:36085/ using document root <none> and password file <none>
I20260806 02:50:46.129522 29640 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:46.129585 29640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:46.129769 29640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:46.132020 29640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "01bd670ca2af4dae9ec72962c4ceec53"
format_stamp: "Formatted at 2026-08-06 02:50:46 on dist-test-slave-3kk6"
server_key: "4d1eadb48c516093e71794cf56f42e08"
server_key_iv: "03ddbda8e4f280811061981c36918226"
server_key_version: "encryptionkey@0"
I20260806 02:50:46.132591 29640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "01bd670ca2af4dae9ec72962c4ceec53"
format_stamp: "Formatted at 2026-08-06 02:50:46 on dist-test-slave-3kk6"
server_key: "4d1eadb48c516093e71794cf56f42e08"
server_key_iv: "03ddbda8e4f280811061981c36918226"
server_key_version: "encryptionkey@0"
I20260806 02:50:46.136512 29640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:46.139140 29658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:46.140575 29640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:46.140799 29640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "01bd670ca2af4dae9ec72962c4ceec53"
format_stamp: "Formatted at 2026-08-06 02:50:46 on dist-test-slave-3kk6"
server_key: "4d1eadb48c516093e71794cf56f42e08"
server_key_iv: "03ddbda8e4f280811061981c36918226"
server_key_version: "encryptionkey@0"
I20260806 02:50:46.140961 29640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:46.158391 29640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:46.162294 29640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:46.162539 29640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:46.163264 29640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:46.164177 29640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:46.164252 29640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:46.164330 29640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:46.164382 29640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:46.173363 29640 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37863
I20260806 02:50:46.173435 29819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37863 every 8 connection(s)
I20260806 02:50:46.174337 29640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:46.183095 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 29640
I20260806 02:50:46.183259 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:46.183589 25098 external_mini_cluster.cc:1428] Setting key 6734879ea67b4ab9cd3dbee57cde0422
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[29292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:46.189621 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.176352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:33437 (local address 127.24.130.190:36799)
0806 02:50:46.176556 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:46.176561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:46.177496 (+ 935us) server_negotiation.cc:407] Connection header received
0806 02:50:46.178932 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:46.178936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:46.178988 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:46.179071 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:46.180306 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.181001 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.182171 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.182401 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.182488 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.185458 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:46.185461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:46.185464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:46.185487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:46.187166 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.187741 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.187747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.187749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.187862 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.188208 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.188212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.188217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.188484 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:46.188607 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:46.189376 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 02:50:46.189492 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:50:46.191048 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.176829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36799 (local address 127.24.130.130:33437)
0806 02:50:46.177309 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:46.177365 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:46.178637 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:46.179244 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:46.179251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:46.179650 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:46.180152 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:46.180166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.181124 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.181127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:46.182031 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:46.182040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.182544 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.182547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:46.182766 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.183489 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:46.183506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:46.185280 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:46.187329 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:46.187336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:46.187355 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:46.187630 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:46.187969 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:46.187972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:46.187973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:46.188113 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:46.188641 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:46.188647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:46.189029 (+ 382us) client_negotiation.cc:769] Sending connection context
0806 02:50:46.189326 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 02:50:46.189746 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260806 02:50:46.192420 29821 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36799
I20260806 02:50:46.192734 29821 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:46.193418 29821 heartbeater.cc:511] Master 127.24.130.190:36799 requested a full tablet report, sending...
I20260806 02:50:46.194796 29356 ts_manager.cc:194] Registered new tserver with Master: 01bd670ca2af4dae9ec72962c4ceec53 (127.24.130.130:37863)
I20260806 02:50:46.195544 29356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:33437
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:46.217778 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36799
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:46.337218 29828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:46.337579 29828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:46.337666 29828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:46.341872 29828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:46.341997 29828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:46.342103 29828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:46.347929 29828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:46.349189 29828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:46.350327 29828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:46.361088 29840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:46.361099 29836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:46.361088 29835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:46.361378 29828 server_base.cc:1034] running on GCE node
I20260806 02:50:46.361994 29828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:46.362818 29828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:46.364007 29828 hybrid_clock.cc:648] HybridClock initialized: now 1785984646363996 us; error 33 us; skew 500 ppm
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[29292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:46.367252 29828 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:46.368228 29828 webserver.cc:460] Webserver started at http://127.24.130.131:38369/ using document root <none> and password file <none>
I20260806 02:50:46.368834 29828 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:46.368897 29828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:46.369083 29828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:46.371425 29828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "e2afc3a0e4b84b35a1ba6636964571b6"
format_stamp: "Formatted at 2026-08-06 02:50:46 on dist-test-slave-3kk6"
server_key: "32511718216286b5109072029da9d3b4"
server_key_iv: "277205dbbedfe9896ffe1fe19819c714"
server_key_version: "encryptionkey@0"
I20260806 02:50:46.372030 29828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "e2afc3a0e4b84b35a1ba6636964571b6"
format_stamp: "Formatted at 2026-08-06 02:50:46 on dist-test-slave-3kk6"
server_key: "32511718216286b5109072029da9d3b4"
server_key_iv: "277205dbbedfe9896ffe1fe19819c714"
server_key_version: "encryptionkey@0"
I20260806 02:50:46.376324 29828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:46.379418 29853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:46.381006 29828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:46.381161 29828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "e2afc3a0e4b84b35a1ba6636964571b6"
format_stamp: "Formatted at 2026-08-06 02:50:46 on dist-test-slave-3kk6"
server_key: "32511718216286b5109072029da9d3b4"
server_key_iv: "277205dbbedfe9896ffe1fe19819c714"
server_key_version: "encryptionkey@0"
I20260806 02:50:46.381313 29828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:46.392617 29828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:46.396590 29828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:46.396833 29828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:46.397534 29828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:46.398486 29828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:46.398535 29828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:46.398609 29828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:46.398748 29828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 02:50:46.409574 29828 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:36669
I20260806 02:50:46.409595 30013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:36669 every 8 connection(s)
I20260806 02:50:46.410609 29828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:46.416581 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 29828
I20260806 02:50:46.416710 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:46.417024 25098 external_mini_cluster.cc:1428] Setting key 187b3d320b48ac9f3aba5828b783f99e
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[29292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:46.426819 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.412800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:46109 (local address 127.24.130.190:36799)
0806 02:50:46.412955 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:46.412959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:46.413907 (+ 948us) server_negotiation.cc:407] Connection header received
0806 02:50:46.415585 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:46.415588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:46.415644 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:46.415726 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:46.417252 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.417929 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.419605 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.419833 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.419922 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.422811 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:46.422815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:46.422817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:46.422844 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:46.424488 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.425079 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.425082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.425084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.425214 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.425587 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.425590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.425591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.425801 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:46.425882 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:46.426476 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 02:50:46.426597 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:50:46.427749 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.413110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36799 (local address 127.24.130.131:46109)
0806 02:50:46.413665 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:46.413715 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:46.415320 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:46.415933 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:46.415941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:46.416382 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:46.417034 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:46.417050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.418077 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.418081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:46.419491 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:46.419502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.419995 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.419998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:46.420149 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.420776 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:46.420793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:46.422554 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:46.424622 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:46.424627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:46.424642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:46.424959 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:46.425323 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:46.425325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:46.425327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:46.425488 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:46.425909 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:46.425915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:46.426202 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 02:50:46.426371 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:50:46.426733 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":134,"threads_started":1}
I20260806 02:50:46.429193 30015 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36799
I20260806 02:50:46.429523 30015 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:46.430197 30015 heartbeater.cc:511] Master 127.24.130.190:36799 requested a full tablet report, sending...
I20260806 02:50:46.431610 29355 ts_manager.cc:194] Registered new tserver with Master: e2afc3a0e4b84b35a1ba6636964571b6 (127.24.130.131:36669)
I20260806 02:50:46.432288 29355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:46109
I20260806 02:50:46.444486 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[29292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984646, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[29292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984646, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:46.475157 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.465134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.24.130.190:36799)
0806 02:50:46.465350 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:46.465356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:46.465493 (+ 137us) server_negotiation.cc:407] Connection header received
0806 02:50:46.465679 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:46.465685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:46.465739 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:46.465834 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:46.466635 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.467313 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.468243 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.468477 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.468596 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.471449 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:46.471453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:46.471456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:46.471496 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:46.473131 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.473615 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.473624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.473625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.473720 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.474035 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.474039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.474040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.474345 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:46.474504 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:46.474832 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:50:46.474966 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260806 02:50:46.475687 29355 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43986
I20260806 02:50:46.476816 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 29442
I20260806 02:50:46.501467 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 29640
I20260806 02:50:46.523162 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 29828
I20260806 02:50:46.538048 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 29315
2026-08-06T02:50:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2494 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[30032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[30032](info): set up 2 sockets
Aug 06 02:50:46 dist-test-slave-3kk6 krb5kdc[30032](info): commencing operation
krb5kdc: starting...
W20260806 02:50:47.764474 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:47 dist-test-slave-3kk6 krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:47.904121 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36911
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:48.023790 30050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:48.024158 30050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:48.024233 30050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:48.028409 30050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:48.028517 30050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:48.028553 30050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:48.028592 30050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:48.028630 30050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:48.034282 30050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:48.035586 30050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:48.036726 30050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:48.046154 30057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.046154 30056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.046249 30050 server_base.cc:1034] running on GCE node
W20260806 02:50:48.046155 30060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.047293 30050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:48.048486 30050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:48.049650 30050 hybrid_clock.cc:648] HybridClock initialized: now 1785984648049623 us; error 48 us; skew 500 ppm
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:48.052735 30050 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:48.053726 30050 webserver.cc:460] Webserver started at http://127.24.130.190:38121/ using document root <none> and password file <none>
I20260806 02:50:48.054349 30050 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:48.054450 30050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:48.054760 30050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:48.057032 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "a6ae8a6aeac142219aac91a72b468928"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "6905fdcd79b2451cb2d9a1004585e7ce"
server_key_iv: "f549ac3d394162992eeb039937498742"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.057693 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "a6ae8a6aeac142219aac91a72b468928"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "6905fdcd79b2451cb2d9a1004585e7ce"
server_key_iv: "f549ac3d394162992eeb039937498742"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.061897 30050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:48.064770 30070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.066219 30050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:48.066397 30050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "a6ae8a6aeac142219aac91a72b468928"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "6905fdcd79b2451cb2d9a1004585e7ce"
server_key_iv: "f549ac3d394162992eeb039937498742"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.066540 30050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:48.081034 30050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:48.085021 30050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:48.085268 30050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:48.092924 30145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:36911 every 8 connection(s)
I20260806 02:50:48.092957 30050 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:36911
I20260806 02:50:48.094420 30050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:48.098906 30147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:48.102463 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 30050
I20260806 02:50:48.102614 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:48.102954 25098 external_mini_cluster.cc:1428] Setting key 432fd7e753986f3698f38b2a6fafcde4
I20260806 02:50:48.107741 30147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928: Bootstrap starting.
I20260806 02:50:48.110615 30147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:48.111712 30147 log.cc:824] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:48.113924 30147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928: No bootstrap required, opened a new log
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:48.118522 30147 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae8a6aeac142219aac91a72b468928" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36911 } }
I20260806 02:50:48.118855 30147 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:48.118919 30147 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ae8a6aeac142219aac91a72b468928, State: Initialized, Role: FOLLOWER
I20260806 02:50:48.119428 30147 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae8a6aeac142219aac91a72b468928" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36911 } }
I20260806 02:50:48.119620 30147 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:48.119695 30147 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:48.119782 30147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:48.121029 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.105088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.24.130.190:36911)
0806 02:50:48.105664 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.105685 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.105733 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:50:48.106606 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.106633 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.107119 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.107526 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.108428 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.109949 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.110924 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.111716 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.111948 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.114844 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.114851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.114868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:48.114903 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.117531 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.117998 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.118004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.118010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.118156 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.118432 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.118435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.118437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.119035 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.119272 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.119635 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.119974 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":213,"threads_started":1}
I20260806 02:50:48.121039 30147 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae8a6aeac142219aac91a72b468928" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36911 } }
I20260806 02:50:48.121433 30147 leader_election.cc:302] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ae8a6aeac142219aac91a72b468928; no voters:
I20260806 02:50:48.121801 30147 leader_election.cc:288] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:48.121954 30155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:48.122241 30155 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [term 1 LEADER]: Becoming Leader. State: Replica: a6ae8a6aeac142219aac91a72b468928, State: Running, Role: LEADER
I20260806 02:50:48.122575 30155 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae8a6aeac142219aac91a72b468928" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36911 } }
I20260806 02:50:48.123129 30147 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:48.123816 30155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ae8a6aeac142219aac91a72b468928. Latest consensus state: current_term: 1 leader_uuid: "a6ae8a6aeac142219aac91a72b468928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae8a6aeac142219aac91a72b468928" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36911 } } }
I20260806 02:50:48.123936 30155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:48.123906 30157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ae8a6aeac142219aac91a72b468928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ae8a6aeac142219aac91a72b468928" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36911 } } }
I20260806 02:50:48.123980 30157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:48.124459 30164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:48.127635 30164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:48.133653 30164 catalog_manager.cc:1347] Generated new cluster ID: 03439f1854c64032b9f8c04a0de422c1
I20260806 02:50:48.133752 30164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:48.153191 30164 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:48.154464 30164 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:48.162415 30164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ae8a6aeac142219aac91a72b468928: Generated new TSK 0
I20260806 02:50:48.163261 30164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:48.198196 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36911
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:48.325354 30185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:48.325660 30185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:48.325738 30185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:48.330008 30185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:48.330091 30185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:48.330189 30185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:48.336123 30185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:48.337376 30185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:48.338482 30185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:48.348847 30193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.348802 30191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.348815 30190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.349261 30185 server_base.cc:1034] running on GCE node
I20260806 02:50:48.349763 30185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:48.350482 30185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:48.351701 30185 hybrid_clock.cc:648] HybridClock initialized: now 1785984648351676 us; error 45 us; skew 500 ppm
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:48.354923 30185 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:48.356112 30185 webserver.cc:460] Webserver started at http://127.24.130.129:36007/ using document root <none> and password file <none>
I20260806 02:50:48.356750 30185 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:48.356853 30185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:48.357121 30185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:48.359541 30185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "21451f6fa20145b8bbb673476b902920"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "41941cf48ce533049312877ad8a4c533"
server_key_iv: "0e7f0a533c81d9819e7a42ce5a7c97ac"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.360184 30185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "21451f6fa20145b8bbb673476b902920"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "41941cf48ce533049312877ad8a4c533"
server_key_iv: "0e7f0a533c81d9819e7a42ce5a7c97ac"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.364470 30185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:48.367311 30204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.368657 30185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:48.368810 30185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "21451f6fa20145b8bbb673476b902920"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "41941cf48ce533049312877ad8a4c533"
server_key_iv: "0e7f0a533c81d9819e7a42ce5a7c97ac"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.368949 30185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:48.387190 30185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:48.391040 30185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:48.391280 30185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:48.391968 30185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:48.392915 30185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:48.392990 30185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.393067 30185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:48.393124 30185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.403301 30185 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:40021
I20260806 02:50:48.403321 30380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:40021 every 8 connection(s)
I20260806 02:50:48.404295 30185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:48.407995 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 30185
I20260806 02:50:48.408143 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:48.408478 25098 external_mini_cluster.cc:1428] Setting key 6bbe36dea6cf192eb938ad50f28eef19
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:50:48.419539 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.406214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:44619 (local address 127.24.130.190:36911)
0806 02:50:48.406380 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.406385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.407365 (+ 980us) server_negotiation.cc:407] Connection header received
0806 02:50:48.408962 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.408966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.409029 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.409138 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.410389 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.411012 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.412218 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.412463 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.412584 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.415566 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.415571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.415579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:48.415606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.417181 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.417756 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.417765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.417766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.417848 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.418239 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.418242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.418244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.418453 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.418568 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.419203 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.419350 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:50:48.420559 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.406703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36911 (local address 127.24.130.129:44619)
0806 02:50:48.407192 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.407242 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.408708 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.409333 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.409339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.409721 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:48.410232 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.410246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.411155 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.411158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.412079 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.412089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.412633 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.412637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.412813 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.413496 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:48.413514 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:48.415372 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:48.417321 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:48.417327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:48.417345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:48.417636 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:48.417955 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:48.417959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:48.417961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:48.418125 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:48.418569 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:48.418577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:48.418945 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.419161 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.419529 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:48.422029 30382 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36911
I20260806 02:50:48.422320 30382 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:48.423049 30382 heartbeater.cc:511] Master 127.24.130.190:36911 requested a full tablet report, sending...
I20260806 02:50:48.424894 30087 ts_manager.cc:194] Registered new tserver with Master: 21451f6fa20145b8bbb673476b902920 (127.24.130.129:40021)
I20260806 02:50:48.426102 30087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:44619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:48.441697 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36911
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:48.564062 30391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:48.564396 30391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:48.564445 30391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:48.568694 30391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:48.568799 30391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:48.568864 30391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:48.575165 30391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:48.576365 30391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:48.577411 30391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:48.588313 30399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.588328 30402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.588315 30398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.588680 30391 server_base.cc:1034] running on GCE node
I20260806 02:50:48.589322 30391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:48.590142 30391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:48.591341 30391 hybrid_clock.cc:648] HybridClock initialized: now 1785984648591302 us; error 59 us; skew 500 ppm
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:48.594805 30391 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:48.595913 30391 webserver.cc:460] Webserver started at http://127.24.130.130:43615/ using document root <none> and password file <none>
I20260806 02:50:48.596570 30391 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:48.596680 30391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:48.596946 30391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:48.599393 30391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "57d5b6941be645f89bd5b7ac3f856ac4"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "b4486a0517a1b6e43cd1e2227739b702"
server_key_iv: "b436acef6193e9118ad542aad186c4da"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.600102 30391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "57d5b6941be645f89bd5b7ac3f856ac4"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "b4486a0517a1b6e43cd1e2227739b702"
server_key_iv: "b436acef6193e9118ad542aad186c4da"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.604506 30391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:48.607434 30412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.608846 30391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:48.609016 30391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "57d5b6941be645f89bd5b7ac3f856ac4"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "b4486a0517a1b6e43cd1e2227739b702"
server_key_iv: "b436acef6193e9118ad542aad186c4da"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.609189 30391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:48.625412 30391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:48.629132 30391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:48.629372 30391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:48.630023 30391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:48.631053 30391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:48.631129 30391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.631207 30391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:48.631260 30391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.641323 30391 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:38097
I20260806 02:50:48.641341 30578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:38097 every 8 connection(s)
I20260806 02:50:48.642341 30391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:48.651074 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 30391
I20260806 02:50:48.651221 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:48.651577 25098 external_mini_cluster.cc:1428] Setting key 9e62402f3d8b9cce16fbc8085d139d28
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:48.657923 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.644543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:42443 (local address 127.24.130.190:36911)
0806 02:50:48.644728 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.644733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.645710 (+ 977us) server_negotiation.cc:407] Connection header received
0806 02:50:48.647238 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.647241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.647292 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.647374 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.648700 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.649258 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.650407 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.650630 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.650800 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.653920 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.653927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.653932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:48.653968 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.655664 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.656248 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.656257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.656258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.656341 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.656713 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.656718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.656720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.656929 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.657043 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.657600 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.657739 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:50:48.659201 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.644985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36911 (local address 127.24.130.130:42443)
0806 02:50:48.645528 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.645577 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.647003 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.647547 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.647554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.647950 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:48.648492 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.648507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.649388 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.649391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.650278 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.650287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.650891 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.650897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.651112 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.651944 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:48.651961 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:48.653733 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:48.655816 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:48.655821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:48.655838 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:48.656128 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:48.656454 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:48.656457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:48.656459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:48.656612 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:48.657041 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:48.657051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:48.657340 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.657554 (+ 214us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.657928 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":138,"threads_started":1}
I20260806 02:50:48.660707 30579 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36911
I20260806 02:50:48.661031 30579 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:48.661737 30579 heartbeater.cc:511] Master 127.24.130.190:36911 requested a full tablet report, sending...
I20260806 02:50:48.663101 30087 ts_manager.cc:194] Registered new tserver with Master: 57d5b6941be645f89bd5b7ac3f856ac4 (127.24.130.130:38097)
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:50:48.663925 30087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:42443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:48.685796 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36911
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:48.808547 30585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:48.808866 30585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:48.808912 30585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:48.813112 30585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:48.813189 30585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:48.813252 30585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:48.818789 30585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:48.819897 30585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:48.820941 30585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:48.831137 30593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.831144 30599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.831144 30596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.831416 30585 server_base.cc:1034] running on GCE node
I20260806 02:50:48.831887 30585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:48.832602 30585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:48.833807 30585 hybrid_clock.cc:648] HybridClock initialized: now 1785984648833787 us; error 33 us; skew 500 ppm
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:48.836948 30585 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:48.837926 30585 webserver.cc:460] Webserver started at http://127.24.130.131:46257/ using document root <none> and password file <none>
I20260806 02:50:48.838541 30585 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:48.838644 30585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:48.838902 30585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:48.841672 30585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "09df9e72b3e5406b87853213127ab8dd"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "bfb6830fe6abefc28b1b3cd1f60d7c82"
server_key_iv: "441af033a264d932d308a1355d494460"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.842348 30585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "09df9e72b3e5406b87853213127ab8dd"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "bfb6830fe6abefc28b1b3cd1f60d7c82"
server_key_iv: "441af033a264d932d308a1355d494460"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.846585 30585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:50:48.849206 30608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.850463 30585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:50:48.850618 30585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "09df9e72b3e5406b87853213127ab8dd"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-3kk6"
server_key: "bfb6830fe6abefc28b1b3cd1f60d7c82"
server_key_iv: "441af033a264d932d308a1355d494460"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.850800 30585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:48.870890 30585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:48.874558 30585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:48.874807 30585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:48.875470 30585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:48.876350 30585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:48.876422 30585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.876497 30585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:48.876550 30585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.885627 30585 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:42369
I20260806 02:50:48.885646 30777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:42369 every 8 connection(s)
I20260806 02:50:48.886627 30585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:48.894770 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 30585
I20260806 02:50:48.894955 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:48.895289 25098 external_mini_cluster.cc:1428] Setting key 959ca925cc81c5e8a13116fbdc2756a8
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:48.901670 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.888647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:40549 (local address 127.24.130.190:36911)
0806 02:50:48.888794 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.888798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.889639 (+ 841us) server_negotiation.cc:407] Connection header received
0806 02:50:48.891198 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.891201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.891266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.891348 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.892635 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.893199 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.894410 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.894716 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.894847 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.897938 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.897941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.897943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:48.897970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.899451 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.900032 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.900035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.900036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.900121 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.900455 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.900458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.900459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.900665 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.900751 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.901413 (+ 662us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.901537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:50:48.902690 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.888989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36911 (local address 127.24.130.131:40549)
0806 02:50:48.889473 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.889520 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.890974 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.891527 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.891534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.891916 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:48.892475 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.892490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.893370 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.893373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.894265 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.894274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.894888 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.894892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.895126 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.895847 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:48.895868 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:48.897743 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:48.899589 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:48.899594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:48.899608 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:48.899927 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:48.900217 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:48.900220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:48.900222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:48.900370 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:48.900803 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:48.900809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:48.901068 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.901372 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.901693 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":135,"threads_started":1}
I20260806 02:50:48.903995 30780 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36911
I20260806 02:50:48.904273 30780 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:48.904826 30780 heartbeater.cc:511] Master 127.24.130.190:36911 requested a full tablet report, sending...
I20260806 02:50:48.905969 30087 ts_manager.cc:194] Registered new tserver with Master: 09df9e72b3e5406b87853213127ab8dd (127.24.130.131:42369)
I20260806 02:50:48.906582 30087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:40549
I20260806 02:50:48.909459 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:48.938529 30792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.928515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.24.130.129:40021)
0806 02:50:48.928915 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.928922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.928949 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:50:48.929003 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.929009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.929207 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.929341 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.930122 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.931003 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.931910 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.932300 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.932413 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.935064 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.935074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.935089 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:48.935115 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.936752 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.937177 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.937182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.937184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.937272 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.937516 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.937518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.937519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.937771 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.937956 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.938132 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.938308 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":157,"threads_started":1}
Aug 06 02:50:48 dist-test-slave-3kk6 krb5kdc[30032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:48.950171 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.941591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.24.130.190:36911)
0806 02:50:48.941749 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.941756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.941774 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:50:48.941936 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.941941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.941991 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.942080 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.942756 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.943297 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.944099 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.944331 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.944449 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.946976 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.946979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.946981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:48.947008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.948373 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.948819 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.948827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.948829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.948916 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:48.949206 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:48.949210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:48.949212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:48.949455 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.949602 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.949857 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.949995 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 02:50:48.950702 30087 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50376
I20260806 02:50:48.951429 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 30185
I20260806 02:50:48.973402 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 30391
I20260806 02:50:48.988426 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 30585
I20260806 02:50:49.002271 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 30050
2026-08-06T02:50:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2459 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:49 dist-test-slave-3kk6 krb5kdc[30798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:49 dist-test-slave-3kk6 krb5kdc[30798](info): set up 2 sockets
Aug 06 02:50:49 dist-test-slave-3kk6 krb5kdc[30798](info): commencing operation
krb5kdc: starting...
W20260806 02:50:50.251307 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:50 dist-test-slave-3kk6 krb5kdc[30798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:50.391009 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36155
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:50.512300 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:50.512691 30819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:50.512785 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:50.517206 30819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:50.517333 30819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:50.517369 30819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:50.517406 30819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:50.517444 30819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:50.523212 30819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:50.524466 30819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:50.525583 30819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:50.534838 30827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:50.534857 30828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:50.535030 30819 server_base.cc:1034] running on GCE node
W20260806 02:50:50.534837 30833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:50.535748 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:50.537006 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:50.538192 30819 hybrid_clock.cc:648] HybridClock initialized: now 1785984650538159 us; error 52 us; skew 500 ppm
Aug 06 02:50:50 dist-test-slave-3kk6 krb5kdc[30798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984650, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:50.544859 30819 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:50.545833 30819 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:50.545913 30819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:50.546103 30819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:50.548446 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "5b88ed7e263e4a718923676cc885319f"
format_stamp: "Formatted at 2026-08-06 02:50:50 on dist-test-slave-3kk6"
server_key: "2ba0a51cdd53d813399b88473fbd0ee6"
server_key_iv: "d7ef40dd664fba9d32b719a5cffe706a"
server_key_version: "encryptionkey@0"
I20260806 02:50:50.549154 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "5b88ed7e263e4a718923676cc885319f"
format_stamp: "Formatted at 2026-08-06 02:50:50 on dist-test-slave-3kk6"
server_key: "2ba0a51cdd53d813399b88473fbd0ee6"
server_key_iv: "d7ef40dd664fba9d32b719a5cffe706a"
server_key_version: "encryptionkey@0"
I20260806 02:50:50.553532 30819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:50.556273 30842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:50.557701 30819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:50.557847 30819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "5b88ed7e263e4a718923676cc885319f"
format_stamp: "Formatted at 2026-08-06 02:50:50 on dist-test-slave-3kk6"
server_key: "2ba0a51cdd53d813399b88473fbd0ee6"
server_key_iv: "d7ef40dd664fba9d32b719a5cffe706a"
server_key_version: "encryptionkey@0"
I20260806 02:50:50.557981 30819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:50.569377 30819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:50.573266 30819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:50.573460 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:50.581420 30819 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:36155
I20260806 02:50:50.581712 30928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:36155 every 8 connection(s)
I20260806 02:50:50.582180 30819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:50.585160 30929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:50.589562 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 30819
I20260806 02:50:50.589738 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:50.590067 25098 external_mini_cluster.cc:1428] Setting key 018a8f36f779f23913b1a26d159724cc
I20260806 02:50:50.591813 30929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: Bootstrap starting.
I20260806 02:50:50.594600 30929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:50.595552 30929 log.cc:824] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:50.598242 30929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: No bootstrap required, opened a new log
Aug 06 02:50:50 dist-test-slave-3kk6 krb5kdc[30798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:50.603211 30929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b88ed7e263e4a718923676cc885319f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36155 } }
I20260806 02:50:50.603534 30929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:50.603636 30929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b88ed7e263e4a718923676cc885319f, State: Initialized, Role: FOLLOWER
I20260806 02:50:50.604238 30929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b88ed7e263e4a718923676cc885319f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36155 } }
I20260806 02:50:50.604462 30929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:50.604552 30929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:50.604669 30929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:50.606122 30929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b88ed7e263e4a718923676cc885319f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36155 } }
I20260806 02:50:50.606539 30929 leader_election.cc:302] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b88ed7e263e4a718923676cc885319f; no voters:
I20260806 02:50:50.606971 30929 leader_election.cc:288] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:50.607159 30940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:50.607328 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:50.591768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.24.130.190:36155)
0806 02:50:50.592301 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:50.592320 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:50.592361 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:50:50.593080 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:50.593099 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:50.593466 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:50.593783 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:50.594911 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:50.596097 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:50.597388 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:50.597942 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:50.598090 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:50.601277 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:50.601283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:50.601301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:50.601334 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:50.603427 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:50.604004 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:50.604010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:50.604017 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:50.604177 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:50.604511 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:50.604516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:50.604519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:50.605141 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:50.605374 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:50.605739 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 02:50:50.606129 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":206,"threads_started":1}
I20260806 02:50:50.607432 30940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [term 1 LEADER]: Becoming Leader. State: Replica: 5b88ed7e263e4a718923676cc885319f, State: Running, Role: LEADER
I20260806 02:50:50.608358 30929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:50.608953 30940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b88ed7e263e4a718923676cc885319f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36155 } }
I20260806 02:50:50.610908 30942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b88ed7e263e4a718923676cc885319f. Latest consensus state: current_term: 1 leader_uuid: "5b88ed7e263e4a718923676cc885319f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b88ed7e263e4a718923676cc885319f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36155 } } }
I20260806 02:50:50.610951 30941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b88ed7e263e4a718923676cc885319f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b88ed7e263e4a718923676cc885319f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36155 } } }
I20260806 02:50:50.611341 30942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:50.611346 30941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:50.614024 30960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:50.614118 30960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:50.614198 30961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:50.615023 30961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:50.621197 30961 catalog_manager.cc:1347] Generated new cluster ID: be253a244f7d4651b21a0817a7aec30f
I20260806 02:50:50.621388 30961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:50.661650 30961 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:50.663170 30961 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:50.671766 30961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: Generated new TSK 0
I20260806 02:50:50.672734 30961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:50.710233 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36155
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:50.834463 30973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:50.834915 30973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:50.835018 30973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:50.839233 30973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:50.839344 30973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:50.839445 30973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:50.845095 30973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:50.846336 30973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:50.847489 30973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:50.857647 30981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:50.857684 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:50.857647 30984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:50.857988 30973 server_base.cc:1034] running on GCE node
I20260806 02:50:50.858462 30973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:50.859253 30973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:50.860499 30973 hybrid_clock.cc:648] HybridClock initialized: now 1785984650860456 us; error 52 us; skew 500 ppm
Aug 06 02:50:50 dist-test-slave-3kk6 krb5kdc[30798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984650, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:50.863844 30973 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:50.864682 30973 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:50.864792 30973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:50.865053 30973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:50.867367 30973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "3aae792ffd9f44298e28974bf2325e78"
format_stamp: "Formatted at 2026-08-06 02:50:50 on dist-test-slave-3kk6"
server_key: "97daafc58aae693fb4744f8eae9d9796"
server_key_iv: "f61686b08f6e882b92a57a87aa87664e"
server_key_version: "encryptionkey@0"
I20260806 02:50:50.868005 30973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "3aae792ffd9f44298e28974bf2325e78"
format_stamp: "Formatted at 2026-08-06 02:50:50 on dist-test-slave-3kk6"
server_key: "97daafc58aae693fb4744f8eae9d9796"
server_key_iv: "f61686b08f6e882b92a57a87aa87664e"
server_key_version: "encryptionkey@0"
I20260806 02:50:50.872200 30973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:50.875051 30990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:50.876477 30973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:50:50.876643 30973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "3aae792ffd9f44298e28974bf2325e78"
format_stamp: "Formatted at 2026-08-06 02:50:50 on dist-test-slave-3kk6"
server_key: "97daafc58aae693fb4744f8eae9d9796"
server_key_iv: "f61686b08f6e882b92a57a87aa87664e"
server_key_version: "encryptionkey@0"
I20260806 02:50:50.876787 30973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:50.889837 30973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:50.893800 30973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:50.894054 30973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:50.894805 30973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:50.895766 30973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:50.895843 30973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:50.895918 30973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:50.895973 30973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:50.905547 30973 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:33913
I20260806 02:50:50.905563 31172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:33913 every 8 connection(s)
I20260806 02:50:50.906349 30973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:50.909508 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 30973
I20260806 02:50:50.909664 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:50.910038 25098 external_mini_cluster.cc:1428] Setting key bdf085efa08443159e5e65a484b7bdbc
Aug 06 02:50:50 dist-test-slave-3kk6 krb5kdc[30798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984650, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:50:50.922313 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:50.908424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:41533 (local address 127.24.130.190:36155)
0806 02:50:50.908573 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:50.908577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:50.909649 (+ 1072us) server_negotiation.cc:407] Connection header received
0806 02:50:50.911368 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:50.911372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:50.911431 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:50.911517 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:50.913305 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:50.913951 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:50.915260 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:50.915505 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:50.915598 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:50.918380 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:50.918384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:50.918386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:50.918415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:50.920024 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:50.920610 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:50.920618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:50.920620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:50.920701 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:50.921074 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:50.921079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:50.921081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:50.921287 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:50.921400 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:50.921980 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 02:50:50.922126 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:50:50.923414 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:50.908850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36155 (local address 127.24.130.129:41533)
0806 02:50:50.909448 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:50.909517 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:50.911143 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:50.911784 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:50.911793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:50.912348 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:50.913103 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:50.913128 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:50.914121 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:50.914124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:50.915128 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:50.915138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:50.915709 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:50.915712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:50.915880 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:50.916588 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:50.916605 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:50.918171 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:50.920166 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:50.920173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:50.920193 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:50.920490 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:50.920800 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:50.920803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:50.920805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:50.920973 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:50.921417 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:50.921426 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:50.921711 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 02:50:50.921920 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:50:50.922288 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:50.924788 31173 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36155
I20260806 02:50:50.925060 31173 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:50.925565 31173 heartbeater.cc:511] Master 127.24.130.190:36155 requested a full tablet report, sending...
I20260806 02:50:50.927214 30869 ts_manager.cc:194] Registered new tserver with Master: 3aae792ffd9f44298e28974bf2325e78 (127.24.130.129:33913)
I20260806 02:50:50.928408 30869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:41533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:50.942798 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36155
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:51.064594 31178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:51.064924 31178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:51.064972 31178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:51.069504 31178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:51.069591 31178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:51.069657 31178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:51.075529 31178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:51.076809 31178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:51.077971 31178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:51.088749 31185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.088857 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.088749 31184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:51.089198 31178 server_base.cc:1034] running on GCE node
I20260806 02:50:51.089635 31178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:51.090296 31178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:51.091464 31178 hybrid_clock.cc:648] HybridClock initialized: now 1785984651091437 us; error 44 us; skew 500 ppm
Aug 06 02:50:51 dist-test-slave-3kk6 krb5kdc[30798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:51.094558 31178 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:51.095497 31178 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:51.095571 31178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:51.095760 31178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:51.098075 31178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "9beef2bcc8514a01bd19c693b026dcb8"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "7944f270aad1bf51cff201b4786a3469"
server_key_iv: "678c4e3dbc7e65ece474d28d4a39e392"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.098774 31178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "9beef2bcc8514a01bd19c693b026dcb8"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "7944f270aad1bf51cff201b4786a3469"
server_key_iv: "678c4e3dbc7e65ece474d28d4a39e392"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.102986 31178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:50:51.105556 31197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.107203 31178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:51.107362 31178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "9beef2bcc8514a01bd19c693b026dcb8"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "7944f270aad1bf51cff201b4786a3469"
server_key_iv: "678c4e3dbc7e65ece474d28d4a39e392"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.107497 31178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:51.146034 31178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:51.149580 31178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:51.149832 31178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:51.150513 31178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:51.151513 31178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:51.151587 31178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.151654 31178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:51.151714 31178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.161460 31178 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37067
I20260806 02:50:51.161511 31357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37067 every 8 connection(s)
I20260806 02:50:51.162201 31178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:51.172227 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 31178
I20260806 02:50:51.172353 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:51.172686 25098 external_mini_cluster.cc:1428] Setting key 536ed85a80fb957be5d82b9e52401e43
Aug 06 02:50:51 dist-test-slave-3kk6 krb5kdc[30798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:51.178376 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.164577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:49501 (local address 127.24.130.190:36155)
0806 02:50:51.164776 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.164780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.165766 (+ 986us) server_negotiation.cc:407] Connection header received
0806 02:50:51.167214 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.167218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.167271 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.167368 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:51.168693 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.169225 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.170464 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.170777 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.170871 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.174087 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:51.174091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:51.174092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:51.174119 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:51.175876 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.176595 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.176597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.176599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.176679 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.177095 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.177098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.177100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.177322 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:51.177438 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.178029 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.178178 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:50:51.179347 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.165074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36155 (local address 127.24.130.130:49501)
0806 02:50:51.165590 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.165637 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.166980 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.167595 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.167603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.168022 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:51.168543 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.168557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.169341 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.169344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.170314 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.170325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.170966 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.170972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.171217 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.171975 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:51.171993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:51.173878 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:51.176014 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:51.176024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:51.176047 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:51.176489 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:51.176798 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:51.176801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:51.176803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:51.176986 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:51.177439 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:51.177449 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:51.177754 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.177975 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.178349 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":134,"threads_started":1}
I20260806 02:50:51.180652 31358 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36155
I20260806 02:50:51.180941 31358 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:51.181466 31358 heartbeater.cc:511] Master 127.24.130.190:36155 requested a full tablet report, sending...
I20260806 02:50:51.182785 30869 ts_manager.cc:194] Registered new tserver with Master: 9beef2bcc8514a01bd19c693b026dcb8 (127.24.130.130:37067)
I20260806 02:50:51.183418 30869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:49501
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:51.207077 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36155
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:51.327674 31365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:51.327975 31365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:51.328023 31365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:51.332154 31365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:51.332217 31365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:51.332280 31365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:51.337841 31365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:51.338985 31365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:51.340055 31365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:51.350564 31371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.350572 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.350584 31374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:51.350873 31365 server_base.cc:1034] running on GCE node
I20260806 02:50:51.351346 31365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:51.352083 31365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:51.353276 31365 hybrid_clock.cc:648] HybridClock initialized: now 1785984651353260 us; error 35 us; skew 500 ppm
Aug 06 02:50:51 dist-test-slave-3kk6 krb5kdc[30798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:51.356488 31365 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:51.357363 31365 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:51.357431 31365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:51.357612 31365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:51.359862 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "64e83ed5791f49bcb375bfd17f796027"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "ea17fe078fd9931f3b03b70e3e0d0750"
server_key_iv: "d7c101792c2f7597a041c719098d6528"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.360441 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "64e83ed5791f49bcb375bfd17f796027"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "ea17fe078fd9931f3b03b70e3e0d0750"
server_key_iv: "d7c101792c2f7597a041c719098d6528"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.364667 31365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:51.367465 31383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.368757 31365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:51.368896 31365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "64e83ed5791f49bcb375bfd17f796027"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "ea17fe078fd9931f3b03b70e3e0d0750"
server_key_iv: "d7c101792c2f7597a041c719098d6528"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.369020 31365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:51.386202 31365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:51.389624 31365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:51.389860 31365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:51.390537 31365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:51.391515 31365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:51.391592 31365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.391681 31365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:51.391726 31365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.401536 31365 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:43499
I20260806 02:50:51.401553 31551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:43499 every 8 connection(s)
I20260806 02:50:51.402235 31365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:51.406242 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 31365
I20260806 02:50:51.406414 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:51.406819 25098 external_mini_cluster.cc:1428] Setting key c03dd42da5f3b93511299d2414272d7a
Aug 06 02:50:51 dist-test-slave-3kk6 krb5kdc[30798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:51.417321 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.404225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38039 (local address 127.24.130.190:36155)
0806 02:50:51.404388 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.404392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.405278 (+ 886us) server_negotiation.cc:407] Connection header received
0806 02:50:51.406804 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.406807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.406874 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.406994 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:51.408624 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.409435 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.410784 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.411012 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.411110 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.413929 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:51.413932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:51.413935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:51.413961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:51.415352 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.415901 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.415904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.415905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.415985 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.416286 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.416288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.416289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.416469 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:51.416545 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.417074 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.417188 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:51.418216 31554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.404584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36155 (local address 127.24.130.131:38039)
0806 02:50:51.405103 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.405150 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.406531 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.407238 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.407249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.407769 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:51.408481 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.408505 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.409561 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.409564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.410556 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.410566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.411190 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.411194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.411368 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.412020 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:51.412037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:51.413748 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:51.415474 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:51.415480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:51.415494 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:51.415809 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:51.416064 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:51.416066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:51.416068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:51.416207 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:51.416565 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:51.416571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:51.416857 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.417026 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.417306 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":142,"threads_started":1}
I20260806 02:50:51.419445 31552 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36155
I20260806 02:50:51.419736 31552 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:51.420320 31552 heartbeater.cc:511] Master 127.24.130.190:36155 requested a full tablet report, sending...
I20260806 02:50:51.421458 30869 ts_manager.cc:194] Registered new tserver with Master: 64e83ed5791f49bcb375bfd17f796027 (127.24.130.131:43499)
I20260806 02:50:51.422005 30869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:38039
I20260806 02:50:51.433652 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:51.443885 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.435974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.24.130.190:36155)
0806 02:50:51.436115 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.436118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.436267 (+ 149us) server_negotiation.cc:407] Connection header received
0806 02:50:51.436348 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.436351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.436399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.436488 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:51.437234 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.437790 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.438639 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.438955 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.439055 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.440509 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:51.440513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:51.440515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:51.440540 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:51.442143 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.442628 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.442631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.442633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.442747 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.443006 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.443008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.443010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.443258 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:51.443369 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.443621 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.443746 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:50:51.447235 30869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:51.450383 30869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:51.465904 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.460810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.24.130.131:43499)
0806 02:50:51.461503 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.461514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.461538 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:50:51.461611 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.461616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.461795 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.461946 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:51.462195 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.463212 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.464497 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.465189 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.465335 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.465502 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.465622 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.465726 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":171,"threads_started":1}
I20260806 02:50:51.465915 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.460362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:43499 (local address 127.0.0.1:53414)
0806 02:50:51.461243 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.461258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.461353 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.461967 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.461970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.461988 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:51.462079 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.462085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.463361 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.463365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.464358 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.464369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.465324 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.465328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.465433 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.465471 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.465594 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.465726 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":80,"thread_start_us":108,"threads_started":1}
I20260806 02:50:51.466193 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.459563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37067 (local address 127.0.0.1:51468)
0806 02:50:51.460089 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.460148 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.460297 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.460995 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.461004 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.461029 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:51.461220 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.461228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.462952 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.462956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.464430 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.464442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.465782 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.465788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.465895 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.465917 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.465969 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.466046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":31,"thread_start_us":110,"threads_started":1}
I20260806 02:50:51.466552 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.459574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33913 (local address 127.0.0.1:42402)
0806 02:50:51.460862 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.460879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.460967 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.461803 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.461806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.461824 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:51.461948 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.461958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.463374 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.463377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.464726 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.464842 (+ 116us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.466081 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.466085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.466168 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.466188 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.466262 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.466331 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":220,"spinlock_wait_cycles":10496,"thread_start_us":119,"threads_started":1}
I20260806 02:50:51.467013 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.459700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.24.130.130:37067)
0806 02:50:51.460428 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.460443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.460470 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:50:51.460553 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.460558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.460787 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.461396 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:51.461570 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.462805 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.464799 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.465629 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.466501 (+ 872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.466677 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.466743 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.466856 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":156,"threads_started":1}
I20260806 02:50:51.467195 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.460362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.24.130.129:33913)
0806 02:50:51.461308 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.461317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.461342 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:50:51.461430 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.461436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.461653 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.461812 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:51.462081 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.463223 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.464975 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.465761 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.465872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.466020 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.466926 (+ 906us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.467039 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":123,"threads_started":1}
I20260806 02:50:51.469856 31449 tablet_service.cc:1467] Processing CreateTablet for tablet 958ff14a190e4bea9ed0185527b702ef (DEFAULT_TABLE table=test-table [id=f32024a9433446a3a1320a50fdfc4557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:51.469877 31261 tablet_service.cc:1467] Processing CreateTablet for tablet 958ff14a190e4bea9ed0185527b702ef (DEFAULT_TABLE table=test-table [id=f32024a9433446a3a1320a50fdfc4557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:51.469877 31068 tablet_service.cc:1467] Processing CreateTablet for tablet 958ff14a190e4bea9ed0185527b702ef (DEFAULT_TABLE table=test-table [id=f32024a9433446a3a1320a50fdfc4557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:51.471014 31449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958ff14a190e4bea9ed0185527b702ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:51.471015 31261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958ff14a190e4bea9ed0185527b702ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:51.471015 31068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958ff14a190e4bea9ed0185527b702ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:51.478250 31571 tablet_bootstrap.cc:492] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Bootstrap starting.
I20260806 02:50:51.479563 31572 tablet_bootstrap.cc:492] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: Bootstrap starting.
I20260806 02:50:51.480051 31573 tablet_bootstrap.cc:492] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Bootstrap starting.
I20260806 02:50:51.481078 31571 tablet_bootstrap.cc:654] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:51.481648 31572 tablet_bootstrap.cc:654] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:51.482329 31573 tablet_bootstrap.cc:654] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:51.482419 31572 log.cc:824] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:51.482465 31571 log.cc:824] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:51.483596 31573 log.cc:824] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:51.484817 31572 tablet_bootstrap.cc:492] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: No bootstrap required, opened a new log
I20260806 02:50:51.485028 31572 ts_tablet_manager.cc:1397] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:50:51.485029 31571 tablet_bootstrap.cc:492] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: No bootstrap required, opened a new log
I20260806 02:50:51.485251 31571 ts_tablet_manager.cc:1397] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:50:51.485603 31573 tablet_bootstrap.cc:492] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: No bootstrap required, opened a new log
I20260806 02:50:51.485781 31573 ts_tablet_manager.cc:1397] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:50:51.488365 31571 raft_consensus.cc:348] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.488669 31571 raft_consensus.cc:374] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:51.488726 31571 raft_consensus.cc:729] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e83ed5791f49bcb375bfd17f796027, State: Initialized, Role: FOLLOWER
I20260806 02:50:51.489198 31571 consensus_queue.cc:260] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.489962 31552 heartbeater.cc:503] Master 127.24.130.190:36155 was elected leader, sending a full tablet report...
I20260806 02:50:51.489826 31572 raft_consensus.cc:348] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.490084 31572 raft_consensus.cc:374] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:51.490132 31572 raft_consensus.cc:729] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aae792ffd9f44298e28974bf2325e78, State: Initialized, Role: FOLLOWER
I20260806 02:50:51.490314 31571 ts_tablet_manager.cc:1428] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:50:51.490610 31572 consensus_queue.cc:260] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.489826 31573 raft_consensus.cc:348] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.490998 31573 raft_consensus.cc:374] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:51.491103 31573 raft_consensus.cc:729] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9beef2bcc8514a01bd19c693b026dcb8, State: Initialized, Role: FOLLOWER
I20260806 02:50:51.491642 31572 ts_tablet_manager.cc:1428] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:50:51.491588 31573 consensus_queue.cc:260] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.491750 31173 heartbeater.cc:503] Master 127.24.130.190:36155 was elected leader, sending a full tablet report...
I20260806 02:50:51.492447 31358 heartbeater.cc:503] Master 127.24.130.190:36155 was elected leader, sending a full tablet report...
I20260806 02:50:51.492761 31573 ts_tablet_manager.cc:1428] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:50:51.511916 31581 raft_consensus.cc:482] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:51.512177 31581 raft_consensus.cc:504] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.513470 31581 leader_election.cc:288] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9beef2bcc8514a01bd19c693b026dcb8 (127.24.130.130:37067), 64e83ed5791f49bcb375bfd17f796027 (127.24.130.131:43499)
I20260806 02:50:51.519337 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.513867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:40215 (local address 127.24.130.130:37067)
0806 02:50:51.514027 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.514032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.514249 (+ 217us) server_negotiation.cc:407] Connection header received
0806 02:50:51.514450 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.514454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.514511 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.514603 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:51.515097 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.515909 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.517876 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.518537 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.518707 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.518766 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.519117 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.519205 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:50:51.519340 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.514490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:38461 (local address 127.24.130.131:43499)
0806 02:50:51.514629 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.514633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.514775 (+ 142us) server_negotiation.cc:407] Connection header received
0806 02:50:51.514869 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.514871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.514911 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.514971 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:51.515266 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.515990 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.517876 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.518537 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.518705 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.518767 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.519139 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.519236 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:50:51.519369 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.514270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:43499 (local address 127.24.130.129:38461)
0806 02:50:51.514636 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.514681 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.514782 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.515014 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.515019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.515043 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:51.515153 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.515157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.516094 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.516097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.517705 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.517718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.518706 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.518710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.518890 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.518949 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.519028 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.519149 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":219,"spinlock_wait_cycles":1920,"thread_start_us":144,"threads_started":1}
I20260806 02:50:51.519369 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.513769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37067 (local address 127.24.130.129:40215)
0806 02:50:51.514136 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.514151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:51.514303 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:51.514739 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:51.514745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:51.514777 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:51.514945 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.514970 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.516027 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.516030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.517707 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:51.517718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.518706 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.518710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:51.518889 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.518943 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:50:51.519028 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:50:51.519152 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":283,"spinlock_wait_cycles":1792,"thread_start_us":181,"threads_started":1}
I20260806 02:50:51.520231 31481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958ff14a190e4bea9ed0185527b702ef" candidate_uuid: "3aae792ffd9f44298e28974bf2325e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e83ed5791f49bcb375bfd17f796027" is_pre_election: true
I20260806 02:50:51.520602 31481 raft_consensus.cc:2454] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aae792ffd9f44298e28974bf2325e78 in term 0.
I20260806 02:50:51.520833 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958ff14a190e4bea9ed0185527b702ef" candidate_uuid: "3aae792ffd9f44298e28974bf2325e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9beef2bcc8514a01bd19c693b026dcb8" is_pre_election: true
I20260806 02:50:51.521174 31283 raft_consensus.cc:2454] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aae792ffd9f44298e28974bf2325e78 in term 0.
I20260806 02:50:51.521178 30992 leader_election.cc:302] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aae792ffd9f44298e28974bf2325e78, 64e83ed5791f49bcb375bfd17f796027; no voters:
I20260806 02:50:51.521524 31581 raft_consensus.cc:2781] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:51.521647 31581 raft_consensus.cc:482] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:51.521714 31581 raft_consensus.cc:3037] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:51.523103 31581 raft_consensus.cc:504] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.523552 31581 leader_election.cc:288] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [CANDIDATE]: Term 1 election: Requested vote from peers 9beef2bcc8514a01bd19c693b026dcb8 (127.24.130.130:37067), 64e83ed5791f49bcb375bfd17f796027 (127.24.130.131:43499)
I20260806 02:50:51.523957 31283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958ff14a190e4bea9ed0185527b702ef" candidate_uuid: "3aae792ffd9f44298e28974bf2325e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9beef2bcc8514a01bd19c693b026dcb8"
I20260806 02:50:51.524020 31481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958ff14a190e4bea9ed0185527b702ef" candidate_uuid: "3aae792ffd9f44298e28974bf2325e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e83ed5791f49bcb375bfd17f796027"
I20260806 02:50:51.524106 31283 raft_consensus.cc:3037] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:51.524142 31481 raft_consensus.cc:3037] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:51.525592 31283 raft_consensus.cc:2454] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aae792ffd9f44298e28974bf2325e78 in term 1.
I20260806 02:50:51.525592 31481 raft_consensus.cc:2454] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aae792ffd9f44298e28974bf2325e78 in term 1.
I20260806 02:50:51.525934 30992 leader_election.cc:302] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aae792ffd9f44298e28974bf2325e78, 9beef2bcc8514a01bd19c693b026dcb8; no voters:
I20260806 02:50:51.526114 31581 raft_consensus.cc:2781] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:51.526378 31581 raft_consensus.cc:686] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 1 LEADER]: Becoming Leader. State: Replica: 3aae792ffd9f44298e28974bf2325e78, State: Running, Role: LEADER
I20260806 02:50:51.526788 31581 consensus_queue.cc:237] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } }
I20260806 02:50:51.530169 30868 catalog_manager.cc:5495] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aae792ffd9f44298e28974bf2325e78 (127.24.130.129). New cstate: current_term: 1 leader_uuid: "3aae792ffd9f44298e28974bf2325e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aae792ffd9f44298e28974bf2325e78" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:51.548126 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.544300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.24.130.129:33913)
0806 02:50:51.544455 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.544461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.544482 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:50:51.544581 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.544583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.544639 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.544761 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:51.545097 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.545623 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.546505 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.546802 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.546928 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.547280 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:51.547714 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:51.547867 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.547906 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.547970 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:50:51.554590 31283 raft_consensus.cc:1261] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 1 FOLLOWER]: Refusing update from remote peer 3aae792ffd9f44298e28974bf2325e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:51.554590 31481 raft_consensus.cc:1261] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 1 FOLLOWER]: Refusing update from remote peer 3aae792ffd9f44298e28974bf2325e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:51.555528 31586 consensus_queue.cc:1035] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64e83ed5791f49bcb375bfd17f796027" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:51.555756 31581 consensus_queue.cc:1035] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9beef2bcc8514a01bd19c693b026dcb8" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:51.564461 31592 mvcc.cc:205] Tried to move back new op lower bound from 7315393132759015424 to 7315393132656762880. Current Snapshot: MvccSnapshot[applied={T|T < 7315393132759015424}]
I20260806 02:50:51.573577 30868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:51.573755 30868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:51.576468 30868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b88ed7e263e4a718923676cc885319f: Sending DeleteTablet for 3 replicas of tablet 958ff14a190e4bea9ed0185527b702ef
I20260806 02:50:51.577435 31068 tablet_service.cc:1514] Processing DeleteTablet for tablet 958ff14a190e4bea9ed0185527b702ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:51 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:42402
I20260806 02:50:51.577654 31261 tablet_service.cc:1514] Processing DeleteTablet for tablet 958ff14a190e4bea9ed0185527b702ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:51 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:51468
I20260806 02:50:51.577657 31449 tablet_service.cc:1514] Processing DeleteTablet for tablet 958ff14a190e4bea9ed0185527b702ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:51 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:53414
I20260806 02:50:51.578166 31604 tablet_replica.cc:330] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78: stopping tablet replica
I20260806 02:50:51.578457 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 30973
I20260806 02:50:51.578539 31604 raft_consensus.cc:2229] T 958ff14a190e4bea9ed0185527b702ef P 3aae792ffd9f44298e28974bf2325e78 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:51.587208 31609 tablet_replica.cc:330] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: stopping tablet replica
I20260806 02:50:51.587631 31609 raft_consensus.cc:2229] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:51.588014 31609 raft_consensus.cc:2258] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:51.588467 31607 tablet_replica.cc:330] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: stopping tablet replica
I20260806 02:50:51.588775 31607 raft_consensus.cc:2229] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:51.589075 31607 raft_consensus.cc:2258] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:51.590180 31609 ts_tablet_manager.cc:1905] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:51.590430 31607 ts_tablet_manager.cc:1905] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:51.593524 31607 ts_tablet_manager.cc:1918] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:51.593641 31607 log.cc:1196] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/958ff14a190e4bea9ed0185527b702ef
I20260806 02:50:51.594015 31607 ts_tablet_manager.cc:1939] T 958ff14a190e4bea9ed0185527b702ef P 9beef2bcc8514a01bd19c693b026dcb8: Deleting consensus metadata
I20260806 02:50:51.594293 31609 ts_tablet_manager.cc:1918] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:51.594501 31609 log.cc:1196] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/958ff14a190e4bea9ed0185527b702ef
I20260806 02:50:51.594987 31609 ts_tablet_manager.cc:1939] T 958ff14a190e4bea9ed0185527b702ef P 64e83ed5791f49bcb375bfd17f796027: Deleting consensus metadata
W20260806 02:50:51.595278 30845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:33913 (error 108)
I20260806 02:50:51.595932 30844 catalog_manager.cc:4841] TS 9beef2bcc8514a01bd19c693b026dcb8 (127.24.130.130:37067): tablet 958ff14a190e4bea9ed0185527b702ef (table test-table [id=f32024a9433446a3a1320a50fdfc4557]) successfully deleted
I20260806 02:50:51.596583 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 31178
I20260806 02:50:51.596612 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.596316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33913 (local address 127.0.0.1:42410)
0806 02:50:51.596458 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:51.596528 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:50:51.600660 30844 catalog_manager.cc:4841] TS 64e83ed5791f49bcb375bfd17f796027 (127.24.130.131:43499): tablet 958ff14a190e4bea9ed0185527b702ef (table test-table [id=f32024a9433446a3a1320a50fdfc4557]) successfully deleted
W20260806 02:50:51.602900 30845 catalog_manager.cc:4584] TS 3aae792ffd9f44298e28974bf2325e78 (127.24.130.129:33913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 958ff14a190e4bea9ed0185527b702ef: Network error: Client connection negotiation failed: client connection to 127.24.130.129:33913: connect: Connection refused (error 111)
I20260806 02:50:51.609072 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 31365
I20260806 02:50:51.623278 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 30819
2026-08-06T02:50:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2618 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:51Z Disabled control of system clock
I20260806 02:50:51.659847 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43699
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:50:51.781845 31616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:51.782189 31616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:51.782230 31616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:51.786613 31616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:51.786787 31616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:51.786808 31616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:51.786857 31616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:51.786888 31616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:51.792204 31616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:51.793536 31616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:51.795179 31616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:51.804587 31616 server_base.cc:1034] running on GCE node
W20260806 02:50:51.804523 31623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.804502 31627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.804522 31622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:51.805536 31616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:51.806951 31616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:51.808195 31616 hybrid_clock.cc:648] HybridClock initialized: now 1785984651808176 us; error 37 us; skew 500 ppm
I20260806 02:50:51.810401 31616 webserver.cc:460] Webserver started at http://127.24.130.190:41093/ using document root <none> and password file <none>
I20260806 02:50:51.811092 31616 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:51.811165 31616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:51.811436 31616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:51.813653 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "09a39c1c34284aa499d38398e3b1261a"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "0b7ce46dbcc4c1af51616a2739b7426b"
server_key_iv: "e60343918f1c4d38c5db274e3c57db52"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.814297 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "09a39c1c34284aa499d38398e3b1261a"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "0b7ce46dbcc4c1af51616a2739b7426b"
server_key_iv: "e60343918f1c4d38c5db274e3c57db52"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.818246 31616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:50:51.821053 31638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.822520 31616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:51.822662 31616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "09a39c1c34284aa499d38398e3b1261a"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-3kk6"
server_key: "0b7ce46dbcc4c1af51616a2739b7426b"
server_key_iv: "e60343918f1c4d38c5db274e3c57db52"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.822841 31616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:51.827895 31616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:51.828552 31616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:51.828761 31616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:51.836328 31616 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43699
I20260806 02:50:51.836341 31717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43699 every 8 connection(s)
I20260806 02:50:51.837396 31616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:51.840860 31718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:51.846899 31718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a: Bootstrap starting.
I20260806 02:50:51.847466 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 31616
I20260806 02:50:51.847594 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:51.847899 25098 external_mini_cluster.cc:1428] Setting key 2156ce4796eeeb857b4b400d139d6841
I20260806 02:50:51.850097 31718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:51.850967 31718 log.cc:824] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:51.853626 31718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a: No bootstrap required, opened a new log
I20260806 02:50:51.854470 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.849495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.24.130.190:43699)
0806 02:50:51.850329 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.850347 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.850388 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:50:51.851214 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.851235 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.851704 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.852099 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:51.853039 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:51.853044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:51.853055 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:51.853065 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:51.853123 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:51.853140 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:51.853240 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.853383 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.853644 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":154,"threads_started":1}
I20260806 02:50:51.857802 31718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a39c1c34284aa499d38398e3b1261a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43699 } }
I20260806 02:50:51.858088 31718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:51.858173 31718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a39c1c34284aa499d38398e3b1261a, State: Initialized, Role: FOLLOWER
I20260806 02:50:51.858645 31718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a39c1c34284aa499d38398e3b1261a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43699 } }
I20260806 02:50:51.858845 31718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:51.858922 31718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:51.859054 31718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:51.860121 31718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a39c1c34284aa499d38398e3b1261a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43699 } }
I20260806 02:50:51.860492 31718 leader_election.cc:302] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a39c1c34284aa499d38398e3b1261a; no voters:
I20260806 02:50:51.860880 31718 leader_election.cc:288] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:51.861074 31727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:51.861320 31727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [term 1 LEADER]: Becoming Leader. State: Replica: 09a39c1c34284aa499d38398e3b1261a, State: Running, Role: LEADER
I20260806 02:50:51.861655 31727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a39c1c34284aa499d38398e3b1261a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43699 } }
I20260806 02:50:51.862438 31718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:51.863615 31729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a39c1c34284aa499d38398e3b1261a. Latest consensus state: current_term: 1 leader_uuid: "09a39c1c34284aa499d38398e3b1261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a39c1c34284aa499d38398e3b1261a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43699 } } }
I20260806 02:50:51.863773 31729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:51.864059 31728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09a39c1c34284aa499d38398e3b1261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a39c1c34284aa499d38398e3b1261a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43699 } } }
I20260806 02:50:51.864145 31728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:51.864184 31737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:51.867563 31737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:51.874105 31737 catalog_manager.cc:1347] Generated new cluster ID: 76548008e2a2422fb84f0bc2feedff23
I20260806 02:50:51.874205 31737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:51.887573 31737 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:51.887979 31737 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:51.894723 31737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a: Generated new TSK 0
I20260806 02:50:51.895521 31737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:51.904748 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43699
--builtin_ntp_servers=127.24.130.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:50:52.029713 31753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:52.030019 31753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:52.030062 31753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:52.034300 31753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:52.034376 31753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:52.034459 31753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:52.039377 31753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:52.040640 31753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:52.042212 31753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:52.052726 31758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.052732 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.052731 31759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:52.053144 31753 server_base.cc:1034] running on GCE node
I20260806 02:50:52.053632 31753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:52.054405 31753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:52.055605 31753 hybrid_clock.cc:648] HybridClock initialized: now 1785984652055599 us; error 66 us; skew 500 ppm
I20260806 02:50:52.057765 31753 webserver.cc:460] Webserver started at http://127.24.130.129:35825/ using document root <none> and password file <none>
I20260806 02:50:52.058422 31753 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:52.058494 31753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:52.058801 31753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:52.061192 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "851013d7eda34c7d8b037a552d73e167"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "6c0a8edaeb855b3ec721ad1ca0b259f5"
server_key_iv: "c5aa7df9708773fbfb472f6f7acfc5f4"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.061880 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "851013d7eda34c7d8b037a552d73e167"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "6c0a8edaeb855b3ec721ad1ca0b259f5"
server_key_iv: "c5aa7df9708773fbfb472f6f7acfc5f4"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.066865 31753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 02:50:52.069849 31774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.071310 31753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:52.071449 31753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "851013d7eda34c7d8b037a552d73e167"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "6c0a8edaeb855b3ec721ad1ca0b259f5"
server_key_iv: "c5aa7df9708773fbfb472f6f7acfc5f4"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.071617 31753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:52.076788 31753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:52.077507 31753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:52.077718 31753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:52.078351 31753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:52.079443 31753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:52.079492 31753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.079564 31753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:52.079596 31753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.089314 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:33975 every 8 connection(s)
I20260806 02:50:52.089290 31753 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:33975
I20260806 02:50:52.090636 31753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:52.093340 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 31753
I20260806 02:50:52.093492 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:52.093801 25098 external_mini_cluster.cc:1428] Setting key 4620a4f0c1af7114ed0b87368a9873df
I20260806 02:50:52.096469 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43699
--builtin_ntp_servers=127.24.130.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:50:52.098852 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.092791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:40943 (local address 127.24.130.190:43699)
0806 02:50:52.092985 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.092991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.094187 (+ 1196us) server_negotiation.cc:407] Connection header received
0806 02:50:52.096128 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.096131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.096208 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.096359 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.098077 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.098085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.098087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.098149 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.098176 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.098188 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.098321 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.098598 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.098681 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:50:52.100441 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.093190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43699 (local address 127.24.130.129:40943)
0806 02:50:52.093985 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.094043 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.095829 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.096661 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.096673 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.096816 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.097677 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.097679 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.097763 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.097867 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.098302 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.098353 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.098362 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.098734 (+ 372us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.099170 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":152,"threads_started":1}
I20260806 02:50:52.102124 31943 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43699
I20260806 02:50:52.102413 31943 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:52.103025 31943 heartbeater.cc:511] Master 127.24.130.190:43699 requested a full tablet report, sending...
I20260806 02:50:52.104956 31668 ts_manager.cc:194] Registered new tserver with Master: 851013d7eda34c7d8b037a552d73e167 (127.24.130.129:33975)
W20260806 02:50:52.220151 31947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:52.220477 31947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:52.220556 31947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:52.224799 31947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:52.224895 31947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:52.224956 31947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:52.230477 31947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:52.231805 31947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:52.233316 31947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:52.243845 31957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.243845 31956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.243847 31961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:52.244344 31947 server_base.cc:1034] running on GCE node
I20260806 02:50:52.244829 31947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:52.245590 31947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:52.246794 31947 hybrid_clock.cc:648] HybridClock initialized: now 1785984652246765 us; error 58 us; skew 500 ppm
I20260806 02:50:52.249264 31947 webserver.cc:460] Webserver started at http://127.24.130.130:34259/ using document root <none> and password file <none>
I20260806 02:50:52.249924 31947 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:52.250029 31947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:52.250293 31947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:52.252748 31947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "7a073f8dfb9d473c98ac2feea2155b5c"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "e15ed903568efeedbcadee1edb8619b9"
server_key_iv: "c686c975ad9b2334fa2e9a864a283964"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.253419 31947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "7a073f8dfb9d473c98ac2feea2155b5c"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "e15ed903568efeedbcadee1edb8619b9"
server_key_iv: "c686c975ad9b2334fa2e9a864a283964"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.257791 31947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:50:52.260706 31969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.262152 31947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:52.262331 31947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "7a073f8dfb9d473c98ac2feea2155b5c"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "e15ed903568efeedbcadee1edb8619b9"
server_key_iv: "c686c975ad9b2334fa2e9a864a283964"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.262519 31947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:52.268162 31947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:52.268870 31947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:52.269075 31947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:52.269676 31947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:52.270748 31947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:52.270825 31947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.270895 31947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:52.270946 31947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.280747 31947 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:44089
I20260806 02:50:52.280776 32129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:44089 every 8 connection(s)
I20260806 02:50:52.281817 31947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:52.286835 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 31947
I20260806 02:50:52.287002 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:52.287348 25098 external_mini_cluster.cc:1428] Setting key cb74f3297ca4d4c79687c434f1ac3393
I20260806 02:50:52.289058 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.283995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:59115 (local address 127.24.130.190:43699)
0806 02:50:52.284161 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.284165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.285364 (+ 1199us) server_negotiation.cc:407] Connection header received
0806 02:50:52.286880 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.286883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.286936 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.287034 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.288507 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.288510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.288512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.288523 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.288546 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.288554 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.288631 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.288832 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.288900 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:50:52.289789 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43699
--builtin_ntp_servers=127.24.130.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:50:52.291781 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.284414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43699 (local address 127.24.130.130:59115)
0806 02:50:52.285165 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.285213 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.286568 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.287292 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.287302 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.287433 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.288182 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.288184 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.288233 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.288347 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.288651 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.288659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.288666 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.288867 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.290887 (+ 2020us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":128,"threads_started":1}
I20260806 02:50:52.293228 32130 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43699
I20260806 02:50:52.293509 32130 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:52.294027 32130 heartbeater.cc:511] Master 127.24.130.190:43699 requested a full tablet report, sending...
I20260806 02:50:52.295303 31668 ts_manager.cc:194] Registered new tserver with Master: 7a073f8dfb9d473c98ac2feea2155b5c (127.24.130.130:44089)
W20260806 02:50:52.416378 32134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:52.416705 32134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:52.416757 32134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:52.421085 32134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:52.421176 32134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:52.421238 32134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:52.426873 32134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:52.428227 32134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:52.429812 32134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:52.440990 32140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.440990 32141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.440990 32143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:52.442308 32134 server_base.cc:1034] running on GCE node
I20260806 02:50:52.442857 32134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:52.443559 32134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:52.444742 32134 hybrid_clock.cc:648] HybridClock initialized: now 1785984652444724 us; error 35 us; skew 500 ppm
I20260806 02:50:52.446954 32134 webserver.cc:460] Webserver started at http://127.24.130.131:40511/ using document root <none> and password file <none>
I20260806 02:50:52.447569 32134 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:52.447638 32134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:52.447883 32134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:52.450301 32134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "dd6f447ca05c849a4a79c77a79149375"
server_key_iv: "4a4c79d2b6033a79fef157c305a80612"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.450979 32134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "dd6f447ca05c849a4a79c77a79149375"
server_key_iv: "4a4c79d2b6033a79fef157c305a80612"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.455618 32134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:50:52.459021 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.460726 32134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:50:52.460904 32134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "dd6f447ca05c849a4a79c77a79149375"
server_key_iv: "4a4c79d2b6033a79fef157c305a80612"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.461055 32134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:52.467175 32134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:52.467972 32134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:52.468201 32134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:52.468829 32134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:52.469882 32134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:52.469930 32134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.470000 32134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:52.470042 32134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.479883 32134 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:43725
I20260806 02:50:52.479933 32319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:43725 every 8 connection(s)
I20260806 02:50:52.480952 32134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:52.488744 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 32134
I20260806 02:50:52.488798 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.483322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:56289 (local address 127.24.130.190:43699)
0806 02:50:52.483494 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.483498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.485216 (+ 1718us) server_negotiation.cc:407] Connection header received
0806 02:50:52.486732 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.486737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.486792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.486916 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.488254 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.488258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.488261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.488283 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.488306 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.488315 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.488458 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.488587 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.488640 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:50:52.488940 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:52.489275 25098 external_mini_cluster.cc:1428] Setting key f7456e568a76aeb06053ed50533eb95f
I20260806 02:50:52.490001 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.483773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43699 (local address 127.24.130.131:56289)
0806 02:50:52.484984 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.485046 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.486434 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.487115 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.487123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.487226 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.487953 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.487955 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.488000 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.488079 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.488412 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.488422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.488428 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.488644 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.488955 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":162,"threads_started":1}
I20260806 02:50:52.491371 32321 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43699
I20260806 02:50:52.491665 32321 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:52.492202 32321 heartbeater.cc:511] Master 127.24.130.190:43699 requested a full tablet report, sending...
I20260806 02:50:52.493300 31668 ts_manager.cc:194] Registered new tserver with Master: cd6eb1e1cfdb49b2b5be1d9e5873fc02 (127.24.130.131:43725)
I20260806 02:50:52.504498 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:52.508427 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.506752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.24.130.190:43699)
0806 02:50:52.506930 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.506933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.507056 (+ 123us) server_negotiation.cc:407] Connection header received
0806 02:50:52.507172 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.507174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.507213 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.507291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.508043 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.508046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.508048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.508061 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.508084 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.508093 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.508203 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.508272 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.508308 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:50:52.511804 31668 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:52.515018 31668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:52.528553 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.525757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.24.130.129:33975)
0806 02:50:52.526154 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.526165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.526191 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:50:52.526249 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.526258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.526417 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.527127 (+ 710us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.527997 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.528003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.528021 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.528034 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.528071 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.528097 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.528221 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.528312 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.528381 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":164,"threads_started":1}
I20260806 02:50:52.528541 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.525288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33975 (local address 127.0.0.1:56966)
0806 02:50:52.525594 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.525607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.525678 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.526597 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.526604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.526712 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.527819 (+ 1107us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.527821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.527830 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.527879 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.528193 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.528197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.528200 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.528298 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.528378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20260806 02:50:52.528561 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.524601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.24.130.131:43725)
0806 02:50:52.525323 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.525340 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.525374 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:50:52.525594 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.525602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.525825 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.525930 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.527801 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.527808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.527822 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.527836 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.527912 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.527951 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.528043 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.528299 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.528395 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":191,"threads_started":1}
I20260806 02:50:52.528556 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.524661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.24.130.130:44089)
0806 02:50:52.525137 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.525148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.525448 (+ 300us) server_negotiation.cc:407] Connection header received
0806 02:50:52.525601 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.525608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.525818 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.525948 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.527825 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.527832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.527841 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.527877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.527916 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.527952 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.528043 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.528299 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.528388 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":136,"threads_started":1}
I20260806 02:50:52.528951 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.524404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:43725 (local address 127.0.0.1:55332)
0806 02:50:52.525225 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.525302 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.525438 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.526798 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.526801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.526813 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.527479 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.527480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.527499 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.527628 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.528124 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.528135 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.528141 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.528734 (+ 593us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.528782 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":157,"threads_started":1}
I20260806 02:50:52.529225 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.524404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:44089 (local address 127.0.0.1:59948)
0806 02:50:52.525255 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.525302 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.525437 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.526877 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.526882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.526898 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.527643 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.527644 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.527654 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.527679 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.528124 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.528135 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.528141 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.529085 (+ 944us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.529117 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":173,"spinlock_wait_cycles":2944,"thread_start_us":132,"threads_started":1}
I20260806 02:50:52.531857 31846 tablet_service.cc:1467] Processing CreateTablet for tablet 1d041b680b5d4aebaa11a3ea7cdb0897 (DEFAULT_TABLE table=test-table [id=441795c170614b9badd818f1f54cb1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:52.532099 32231 tablet_service.cc:1467] Processing CreateTablet for tablet 1d041b680b5d4aebaa11a3ea7cdb0897 (DEFAULT_TABLE table=test-table [id=441795c170614b9badd818f1f54cb1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:52.532099 32040 tablet_service.cc:1467] Processing CreateTablet for tablet 1d041b680b5d4aebaa11a3ea7cdb0897 (DEFAULT_TABLE table=test-table [id=441795c170614b9badd818f1f54cb1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:52.533018 31846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d041b680b5d4aebaa11a3ea7cdb0897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:52.533165 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d041b680b5d4aebaa11a3ea7cdb0897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:52.533165 32040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d041b680b5d4aebaa11a3ea7cdb0897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:52.540529 32342 tablet_bootstrap.cc:492] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: Bootstrap starting.
I20260806 02:50:52.541481 32343 tablet_bootstrap.cc:492] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Bootstrap starting.
I20260806 02:50:52.542115 32344 tablet_bootstrap.cc:492] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Bootstrap starting.
I20260806 02:50:52.543525 32342 tablet_bootstrap.cc:654] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:52.543675 32343 tablet_bootstrap.cc:654] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:52.544239 32344 tablet_bootstrap.cc:654] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:52.544489 32343 log.cc:824] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:52.544502 32342 log.cc:824] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:52.545077 32344 log.cc:824] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:52.546718 32343 tablet_bootstrap.cc:492] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: No bootstrap required, opened a new log
I20260806 02:50:52.546797 32342 tablet_bootstrap.cc:492] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: No bootstrap required, opened a new log
I20260806 02:50:52.546963 32343 ts_tablet_manager.cc:1397] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260806 02:50:52.546995 32342 ts_tablet_manager.cc:1397] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:50:52.547051 32344 tablet_bootstrap.cc:492] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: No bootstrap required, opened a new log
I20260806 02:50:52.547230 32344 ts_tablet_manager.cc:1397] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:50:52.551187 32342 raft_consensus.cc:348] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.551498 32342 raft_consensus.cc:374] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:52.551558 32342 raft_consensus.cc:729] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851013d7eda34c7d8b037a552d73e167, State: Initialized, Role: FOLLOWER
I20260806 02:50:52.551586 32343 raft_consensus.cc:348] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.551895 32343 raft_consensus.cc:374] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:52.551957 32343 raft_consensus.cc:729] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6eb1e1cfdb49b2b5be1d9e5873fc02, State: Initialized, Role: FOLLOWER
I20260806 02:50:52.552119 32342 consensus_queue.cc:260] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.552405 32343 consensus_queue.cc:260] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.552502 32344 raft_consensus.cc:348] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.552779 32344 raft_consensus.cc:374] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:52.552847 32344 raft_consensus.cc:729] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a073f8dfb9d473c98ac2feea2155b5c, State: Initialized, Role: FOLLOWER
I20260806 02:50:52.553097 31943 heartbeater.cc:503] Master 127.24.130.190:43699 was elected leader, sending a full tablet report...
I20260806 02:50:52.553331 32342 ts_tablet_manager.cc:1428] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:52.553278 32344 consensus_queue.cc:260] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.553440 32343 ts_tablet_manager.cc:1428] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:52.553807 32321 heartbeater.cc:503] Master 127.24.130.190:43699 was elected leader, sending a full tablet report...
I20260806 02:50:52.554201 32130 heartbeater.cc:503] Master 127.24.130.190:43699 was elected leader, sending a full tablet report...
I20260806 02:50:52.554682 32344 ts_tablet_manager.cc:1428] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 02:50:52.592109 31945 tablet.cc:2369] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:52.633638 32352 raft_consensus.cc:482] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:52.633962 32352 raft_consensus.cc:504] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.635275 32352 leader_election.cc:288] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd6eb1e1cfdb49b2b5be1d9e5873fc02 (127.24.130.131:43725), 7a073f8dfb9d473c98ac2feea2155b5c (127.24.130.130:44089)
I20260806 02:50:52.637764 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.635767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:37667 (local address 127.24.130.130:44089)
0806 02:50:52.635919 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.635926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.636140 (+ 214us) server_negotiation.cc:407] Connection header received
0806 02:50:52.636312 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.636317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.636362 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.636454 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.637351 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.637354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.637357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.637370 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.637396 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.637404 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.637507 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.637579 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.637618 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:50:52.637809 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.635626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:44089 (local address 127.24.130.129:37667)
0806 02:50:52.636037 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.636049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.636200 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.636479 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.636482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.636497 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.637146 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.637150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.637165 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.637201 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.637490 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.637493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.637496 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.637576 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.637618 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":52,"spinlock_wait_cycles":9472,"thread_start_us":69,"threads_started":1}
I20260806 02:50:52.637813 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.635803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:42801 (local address 127.24.130.131:43725)
0806 02:50:52.636008 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.636015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.636121 (+ 106us) server_negotiation.cc:407] Connection header received
0806 02:50:52.636324 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.636326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.636359 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.636449 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.637396 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.637399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.637401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.637420 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.637437 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.637445 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.637545 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.637610 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.637651 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260806 02:50:52.638028 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.635626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:43725 (local address 127.24.130.129:42801)
0806 02:50:52.635999 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.636017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:52.636198 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:52.636472 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:52.636475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:52.636490 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:52.637240 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:52.637241 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:52.637251 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:52.637276 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:52.637529 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:52.637532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:52.637535 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:50:52.637618 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:50:52.637651 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"spinlock_wait_cycles":40704,"thread_start_us":100,"threads_started":1}
I20260806 02:50:52.638581 32067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d041b680b5d4aebaa11a3ea7cdb0897" candidate_uuid: "851013d7eda34c7d8b037a552d73e167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" is_pre_election: true
I20260806 02:50:52.638599 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d041b680b5d4aebaa11a3ea7cdb0897" candidate_uuid: "851013d7eda34c7d8b037a552d73e167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" is_pre_election: true
I20260806 02:50:52.638940 32263 raft_consensus.cc:2454] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851013d7eda34c7d8b037a552d73e167 in term 0.
I20260806 02:50:52.638944 32067 raft_consensus.cc:2454] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 851013d7eda34c7d8b037a552d73e167 in term 0.
I20260806 02:50:52.639436 31784 leader_election.cc:302] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a073f8dfb9d473c98ac2feea2155b5c, 851013d7eda34c7d8b037a552d73e167, cd6eb1e1cfdb49b2b5be1d9e5873fc02; no voters:
I20260806 02:50:52.639706 32352 raft_consensus.cc:2781] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:52.639803 32352 raft_consensus.cc:482] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:52.639848 32352 raft_consensus.cc:3037] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:52.640969 32352 raft_consensus.cc:504] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.641340 32352 leader_election.cc:288] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [CANDIDATE]: Term 1 election: Requested vote from peers cd6eb1e1cfdb49b2b5be1d9e5873fc02 (127.24.130.131:43725), 7a073f8dfb9d473c98ac2feea2155b5c (127.24.130.130:44089)
I20260806 02:50:52.641731 32067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d041b680b5d4aebaa11a3ea7cdb0897" candidate_uuid: "851013d7eda34c7d8b037a552d73e167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a073f8dfb9d473c98ac2feea2155b5c"
I20260806 02:50:52.641739 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d041b680b5d4aebaa11a3ea7cdb0897" candidate_uuid: "851013d7eda34c7d8b037a552d73e167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02"
I20260806 02:50:52.641882 32263 raft_consensus.cc:3037] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:52.641855 32067 raft_consensus.cc:3037] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:52.643090 32263 raft_consensus.cc:2454] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851013d7eda34c7d8b037a552d73e167 in term 1.
I20260806 02:50:52.643095 32067 raft_consensus.cc:2454] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 851013d7eda34c7d8b037a552d73e167 in term 1.
I20260806 02:50:52.643410 31784 leader_election.cc:302] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 851013d7eda34c7d8b037a552d73e167, cd6eb1e1cfdb49b2b5be1d9e5873fc02; no voters:
I20260806 02:50:52.643597 32352 raft_consensus.cc:2781] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:52.643857 32352 raft_consensus.cc:686] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [term 1 LEADER]: Becoming Leader. State: Replica: 851013d7eda34c7d8b037a552d73e167, State: Running, Role: LEADER
I20260806 02:50:52.644202 32352 consensus_queue.cc:237] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } }
I20260806 02:50:52.647926 31666 catalog_manager.cc:5495] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 reported cstate change: term changed from 0 to 1, leader changed from <none> to 851013d7eda34c7d8b037a552d73e167 (127.24.130.129). New cstate: current_term: 1 leader_uuid: "851013d7eda34c7d8b037a552d73e167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "851013d7eda34c7d8b037a552d73e167" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33975 } health_report { overall_health: HEALTHY } } }
I20260806 02:50:52.686842 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.685067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.24.130.129:33975)
0806 02:50:52.685227 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:52.685232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:52.685249 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:50:52.685435 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:52.685444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:52.685510 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:52.685609 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:52.686335 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:52.686339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:52.686341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:52.686354 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:52.686381 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:52.686389 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:52.686489 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:52.686575 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 02:50:52.686617 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:50:52.693339 32263 raft_consensus.cc:1261] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 1 FOLLOWER]: Refusing update from remote peer 851013d7eda34c7d8b037a552d73e167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:52.693339 32067 raft_consensus.cc:1261] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 1 FOLLOWER]: Refusing update from remote peer 851013d7eda34c7d8b037a552d73e167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:52.694249 32359 consensus_queue.cc:1035] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6eb1e1cfdb49b2b5be1d9e5873fc02" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:52.694461 32352 consensus_queue.cc:1035] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 851013d7eda34c7d8b037a552d73e167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a073f8dfb9d473c98ac2feea2155b5c" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 44089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:52.702316 32362 mvcc.cc:205] Tried to move back new op lower bound from 7315393137423491072 to 7315393137233854464. Current Snapshot: MvccSnapshot[applied={T|T < 7315393137423491072}]
I20260806 02:50:52.703320 32363 mvcc.cc:205] Tried to move back new op lower bound from 7315393137423491072 to 7315393137233854464. Current Snapshot: MvccSnapshot[applied={T|T < 7315393137423491072}]
I20260806 02:50:52.711782 31666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:52.711962 31666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:52.714732 31666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09a39c1c34284aa499d38398e3b1261a: Sending DeleteTablet for 3 replicas of tablet 1d041b680b5d4aebaa11a3ea7cdb0897
I20260806 02:50:52.715668 32040 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d041b680b5d4aebaa11a3ea7cdb0897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:52 UTC) from {username='slave'} at 127.0.0.1:59948
I20260806 02:50:52.715708 32231 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d041b680b5d4aebaa11a3ea7cdb0897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:52 UTC) from {username='slave'} at 127.0.0.1:55332
I20260806 02:50:52.716308 31846 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d041b680b5d4aebaa11a3ea7cdb0897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:52 UTC) from {username='slave'} at 127.0.0.1:56966
I20260806 02:50:52.716601 32376 tablet_replica.cc:330] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: stopping tablet replica
I20260806 02:50:52.716794 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 31753
I20260806 02:50:52.716974 32376 raft_consensus.cc:2229] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:52.717293 32376 raft_consensus.cc:2258] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:52.717633 32378 tablet_replica.cc:330] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: stopping tablet replica
I20260806 02:50:52.717877 32378 raft_consensus.cc:2229] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:52.718101 32378 raft_consensus.cc:2258] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:52.721614 32378 ts_tablet_manager.cc:1905] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:52.725126 32378 ts_tablet_manager.cc:1918] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:52.725236 32378 log.cc:1196] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/1d041b680b5d4aebaa11a3ea7cdb0897
I20260806 02:50:52.725556 32378 ts_tablet_manager.cc:1939] T 1d041b680b5d4aebaa11a3ea7cdb0897 P cd6eb1e1cfdb49b2b5be1d9e5873fc02: Deleting consensus metadata
I20260806 02:50:52.731704 31647 catalog_manager.cc:4841] TS cd6eb1e1cfdb49b2b5be1d9e5873fc02 (127.24.130.131:43725): tablet 1d041b680b5d4aebaa11a3ea7cdb0897 (table test-table [id=441795c170614b9badd818f1f54cb1f0]) successfully deleted
I20260806 02:50:52.732828 32376 ts_tablet_manager.cc:1905] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:52.736771 32376 ts_tablet_manager.cc:1918] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:52.736918 32376 log.cc:1196] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/1d041b680b5d4aebaa11a3ea7cdb0897
I20260806 02:50:52.737375 32376 ts_tablet_manager.cc:1939] T 1d041b680b5d4aebaa11a3ea7cdb0897 P 7a073f8dfb9d473c98ac2feea2155b5c: Deleting consensus metadata
I20260806 02:50:52.739109 31639 catalog_manager.cc:4841] TS 7a073f8dfb9d473c98ac2feea2155b5c (127.24.130.130:44089): tablet 1d041b680b5d4aebaa11a3ea7cdb0897 (table test-table [id=441795c170614b9badd818f1f54cb1f0]) successfully deleted
W20260806 02:50:52.740461 31647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:33975 (error 108)
I20260806 02:50:52.741432 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 31947
I20260806 02:50:52.741711 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:52.741356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33975 (local address 127.0.0.1:56994)
0806 02:50:52.741494 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:52.741604 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:33975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:50:52.743832 31647 catalog_manager.cc:4584] TS 851013d7eda34c7d8b037a552d73e167 (127.24.130.129:33975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d041b680b5d4aebaa11a3ea7cdb0897: Network error: Client connection negotiation failed: client connection to 127.24.130.129:33975: connect: Connection refused (error 111)
I20260806 02:50:52.754139 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 32134
I20260806 02:50:52.770413 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 31616
2026-08-06T02:50:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1150 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:50:52.799973 25098 mini_oidc.cc:152] Starting JWKS server
I20260806 02:50:52.801872 25098 webserver.cc:460] Webserver started at https://127.0.0.1:43083/ using document root <none> and password file <none>
I20260806 02:50:52.802033 25098 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:50:52.802294 25098 webserver.cc:460] Webserver started at http://127.0.0.1:39389/ using document root <none> and password file <none>
2026-08-06T02:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:52Z Disabled control of system clock
I20260806 02:50:52.814031 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:33247
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:33247
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43083/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/testchainca.cert with env {}
W20260806 02:50:52.930776 32389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:52.931068 32389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:52.931109 32389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:52.935102 32389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:52.935174 32389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:52.935189 32389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:52.935206 32389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:52.935222 32389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:50:52.935238 32389 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:50:52.935251 32389 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43083/jwks.json
I20260806 02:50:52.940757 32389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:33247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43083/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:52.941874 32389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:52.942915 32389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:52.951916 32404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:52.951934 32399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:52.952004 32389 server_base.cc:1034] running on GCE node
W20260806 02:50:52.951957 32398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:52.952816 32389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:52.954048 32389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:52.955250 32389 hybrid_clock.cc:648] HybridClock initialized: now 1785984652955223 us; error 44 us; skew 500 ppm
I20260806 02:50:52.957206 32389 webserver.cc:460] Webserver started at http://127.24.130.190:38877/ using document root <none> and password file <none>
I20260806 02:50:52.957815 32389 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:52.957919 32389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:52.958170 32389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:52.960408 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "81dc051290b14fcf96211fe3771b73c5"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "0883fa7b4e81698439154287e864d3e8"
server_key_iv: "12c8ef28ecdbcc72c861f2885bd43707"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.961089 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "81dc051290b14fcf96211fe3771b73c5"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "0883fa7b4e81698439154287e864d3e8"
server_key_iv: "12c8ef28ecdbcc72c861f2885bd43707"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.965003 32389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:50:52.967629 32413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:52.968912 32389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:52.969079 32389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "81dc051290b14fcf96211fe3771b73c5"
format_stamp: "Formatted at 2026-08-06 02:50:52 on dist-test-slave-3kk6"
server_key: "0883fa7b4e81698439154287e864d3e8"
server_key_iv: "12c8ef28ecdbcc72c861f2885bd43707"
server_key_version: "encryptionkey@0"
I20260806 02:50:52.969209 32389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:52.996914 32389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:52.997713 32389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:52.997974 32389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:53.005651 32389 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:33247
I20260806 02:50:53.005666 32480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:33247 every 8 connection(s)
I20260806 02:50:53.006731 32389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:53.010090 32481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:53.011586 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 32389
I20260806 02:50:53.011754 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:53.012065 25098 external_mini_cluster.cc:1428] Setting key 22a9d05164ab43ae133f68adc24ef9c2
I20260806 02:50:53.016848 32481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5: Bootstrap starting.
I20260806 02:50:53.019133 32481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:53.019922 32481 log.cc:824] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:53.021277 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.013413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.24.130.190:33247)
0806 02:50:53.014046 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.014061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.014099 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:50:53.014834 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.014854 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.015260 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.015562 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:53.015928 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.016861 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.017762 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.018233 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.018423 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.019443 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:53.019449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:53.019461 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:53.019850 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:53.019906 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:53.019927 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:53.020041 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:53.020161 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:50:53.020445 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":149,"threads_started":1}
I20260806 02:50:53.021994 32481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5: No bootstrap required, opened a new log
I20260806 02:50:53.024123 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:53.026105 32481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dc051290b14fcf96211fe3771b73c5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33247 } }
I20260806 02:50:53.026379 32481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:53.026463 32481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dc051290b14fcf96211fe3771b73c5, State: Initialized, Role: FOLLOWER
I20260806 02:50:53.026959 32481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dc051290b14fcf96211fe3771b73c5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33247 } }
I20260806 02:50:53.027119 32481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:53.027210 32481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:53.027324 32481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:53.028280 32481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dc051290b14fcf96211fe3771b73c5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33247 } }
I20260806 02:50:53.028635 32481 leader_election.cc:302] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81dc051290b14fcf96211fe3771b73c5; no voters:
I20260806 02:50:53.028996 32481 leader_election.cc:288] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:53.029119 32487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:53.029398 32487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [term 1 LEADER]: Becoming Leader. State: Replica: 81dc051290b14fcf96211fe3771b73c5, State: Running, Role: LEADER
I20260806 02:50:53.029695 32487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dc051290b14fcf96211fe3771b73c5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33247 } }
I20260806 02:50:53.030339 32481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:53.031358 32490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81dc051290b14fcf96211fe3771b73c5. Latest consensus state: current_term: 1 leader_uuid: "81dc051290b14fcf96211fe3771b73c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dc051290b14fcf96211fe3771b73c5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33247 } } }
I20260806 02:50:53.031503 32490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.032196 32488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81dc051290b14fcf96211fe3771b73c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81dc051290b14fcf96211fe3771b73c5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33247 } } }
I20260806 02:50:53.032387 32488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5 [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:53.035352 32507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:53.035419 32507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:53.035504 32498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:53.036201 32498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:53.041913 32498 catalog_manager.cc:1347] Generated new cluster ID: 36d24e1fa43a475b9e25005ccb7d205c
I20260806 02:50:53.042001 32498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:53.050052 32498 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:53.051098 32498 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:53.057374 32498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81dc051290b14fcf96211fe3771b73c5: Generated new TSK 0
I20260806 02:50:53.058112 32498 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:50:53.092932 32521 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:50:53.093116 32521 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:50:53.093663 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.068560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.24.130.190:33247)
0806 02:50:53.068721 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.068727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.068992 (+ 265us) server_negotiation.cc:407] Connection header received
0806 02:50:53.069213 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.069216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.069273 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.069387 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:50:53.069708 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.070262 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.071544 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.071784 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.071874 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.072276 (+ 402us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:50:53.093117 (+ 20841us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:50:53.093487 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 02:50:53.093777 32484 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40048: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:50:53.095242 32520 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:53.068462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33247 (local address 127.0.0.1:40048)
0806 02:50:53.068867 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:53.068884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:53.069032 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:53.069439 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:53.069443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:53.069452 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:50:53.069589 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:53.069597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.070383 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.070387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:53.071424 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:53.071434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.071916 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.071920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:53.072018 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.072080 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:50:53.093420 (+ 21340us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:50:53.093694 (+ 274us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.130.190:33247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.130.190:33247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:50:53.096087 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 32389
2026-08-06T02:50:53Z chronyd exiting
I20260806 02:50:53.141217 25098 test_util.cc:175] -----------------------------------------------
I20260806 02:50:53.141317 25098 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984624291587-25098-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (343 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:50:53.142499 25098 mini_oidc.cc:152] Starting JWKS server
I20260806 02:50:53.143482 25098 webserver.cc:460] Webserver started at https://127.0.0.1:41237/ using document root <none> and password file <none>
I20260806 02:50:53.143567 25098 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:50:53.143800 25098 webserver.cc:460] Webserver started at http://127.0.0.1:37621/ using document root <none> and password file <none>
2026-08-06T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:53Z Disabled control of system clock
I20260806 02:50:53.154525 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43449
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:34525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41237/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/testchainca.cert with env {}
W20260806 02:50:53.270725 32528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:53.270998 32528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:53.271039 32528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:53.275074 32528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:53.275138 32528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:53.275154 32528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:53.275171 32528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:53.275187 32528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:50:53.275203 32528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:50:53.275218 32528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41237/jwks.json
I20260806 02:50:53.280689 32528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41237/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:53.281759 32528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:53.282793 32528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:53.291838 32536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.291834 32537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.291834 32539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:53.292061 32528 server_base.cc:1034] running on GCE node
I20260806 02:50:53.292773 32528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:53.293879 32528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:53.295044 32528 hybrid_clock.cc:648] HybridClock initialized: now 1785984653295014 us; error 42 us; skew 500 ppm
I20260806 02:50:53.296931 32528 webserver.cc:460] Webserver started at http://127.24.130.190:44595/ using document root <none> and password file <none>
I20260806 02:50:53.297560 32528 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:53.297627 32528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:53.297855 32528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:53.300331 32528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "67fe7af0c3d14a319d97edc7a1786c13"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "80de7ca6eff0d9a4f7eac10caff4f2d1"
server_key_iv: "256e9b0b96d94e82573401ae724d8eac"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.301065 32528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "67fe7af0c3d14a319d97edc7a1786c13"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "80de7ca6eff0d9a4f7eac10caff4f2d1"
server_key_iv: "256e9b0b96d94e82573401ae724d8eac"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.306500 32528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:50:53.310238 32553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:53.311880 32528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:53.312036 32528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "67fe7af0c3d14a319d97edc7a1786c13"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "80de7ca6eff0d9a4f7eac10caff4f2d1"
server_key_iv: "256e9b0b96d94e82573401ae724d8eac"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.312172 32528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:53.322911 32528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:53.323632 32528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:53.323853 32528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:53.331243 32528 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43449
I20260806 02:50:53.331269 32628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43449 every 8 connection(s)
I20260806 02:50:53.332266 32528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:53.335563 32629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:53.341095 32629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13: Bootstrap starting.
I20260806 02:50:53.341846 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 32528
I20260806 02:50:53.341962 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:53.342281 25098 external_mini_cluster.cc:1428] Setting key aaf4568cc5daf38eddc0eb2685ded8fb
I20260806 02:50:53.343978 32629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:53.344871 32629 log.cc:824] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:53.347178 32629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13: No bootstrap required, opened a new log
I20260806 02:50:53.351768 32629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe7af0c3d14a319d97edc7a1786c13" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43449 } }
I20260806 02:50:53.352084 32629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:53.352154 32629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fe7af0c3d14a319d97edc7a1786c13, State: Initialized, Role: FOLLOWER
I20260806 02:50:53.352674 32629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe7af0c3d14a319d97edc7a1786c13" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43449 } }
I20260806 02:50:53.352842 32629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:53.352907 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.343506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.24.130.190:43449)
0806 02:50:53.344292 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.344311 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.344356 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:50:53.345244 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.345269 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.345606 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.345974 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:53.346388 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.347862 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.348761 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.349485 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.349661 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.350537 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:53.350545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:53.350570 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:53.351196 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:53.351274 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:53.351303 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:53.351407 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:53.351544 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 02:50:53.351865 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":138,"threads_started":1}
I20260806 02:50:53.352918 32629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:53.353046 32629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:53.354113 32629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe7af0c3d14a319d97edc7a1786c13" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43449 } }
I20260806 02:50:53.354487 32629 leader_election.cc:302] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67fe7af0c3d14a319d97edc7a1786c13; no voters:
I20260806 02:50:53.354905 32629 leader_election.cc:288] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:53.354976 32635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:53.355194 32635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [term 1 LEADER]: Becoming Leader. State: Replica: 67fe7af0c3d14a319d97edc7a1786c13, State: Running, Role: LEADER
I20260806 02:50:53.355500 32635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe7af0c3d14a319d97edc7a1786c13" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43449 } }
I20260806 02:50:53.355952 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:53.356237 32629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:53.357096 32639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67fe7af0c3d14a319d97edc7a1786c13. Latest consensus state: current_term: 1 leader_uuid: "67fe7af0c3d14a319d97edc7a1786c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe7af0c3d14a319d97edc7a1786c13" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43449 } } }
I20260806 02:50:53.357250 32639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.357592 32637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67fe7af0c3d14a319d97edc7a1786c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe7af0c3d14a319d97edc7a1786c13" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43449 } } }
I20260806 02:50:53.357703 32637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.358196 32651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:53.360999 32651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:53.367208 32651 catalog_manager.cc:1347] Generated new cluster ID: 5c29026b49634e20b51e04a4a976cd56
I20260806 02:50:53.367300 32651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:53.388352 32651 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:53.389456 32651 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:53.408661 32651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67fe7af0c3d14a319d97edc7a1786c13: Generated new TSK 0
I20260806 02:50:53.409546 32651 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:50:53.436614 32680 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:50:53.436733 32680 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:50:53.437434 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.420870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.24.130.190:43449)
0806 02:50:53.421023 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.421028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.421231 (+ 203us) server_negotiation.cc:407] Connection header received
0806 02:50:53.421407 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.421410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.421457 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.421547 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:50:53.421827 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.422746 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.424030 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.424247 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.424319 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.424677 (+ 358us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:50:53.437061 (+ 12384us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:50:53.437319 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 02:50:53.437526 32632 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 02:50:53.437711 32678 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:53.420812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43449 (local address 127.0.0.1:39336)
0806 02:50:53.421132 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:53.421148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:53.421271 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:53.421572 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:53.421576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:53.421584 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:50:53.421703 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:53.421712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.422881 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.422885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:53.423900 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:53.423912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.424364 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.424368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:53.424468 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.424498 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:50:53.437298 (+ 12800us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:50:53.437513 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.130.190:43449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":121,"threads_started":1}
I20260806 02:50:53.438374 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 32528
2026-08-06T02:50:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:50:53.484004 25098 mini_oidc.cc:152] Starting JWKS server
I20260806 02:50:53.484934 25098 webserver.cc:460] Webserver started at https://127.0.0.1:46617/ using document root <none> and password file <none>
I20260806 02:50:53.485071 25098 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:50:53.485370 25098 webserver.cc:460] Webserver started at http://127.0.0.1:43265/ using document root <none> and password file <none>
2026-08-06T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:53Z Disabled control of system clock
I20260806 02:50:53.496192 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:35581
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:35581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46617/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:50:53.612035 32687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:53.612322 32687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:53.612398 32687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:53.616448 32687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:53.616511 32687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:53.616528 32687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:53.616544 32687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:53.616559 32687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:50:53.616616 32687 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:50:53.616643 32687 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46617/jwks.json
I20260806 02:50:53.621445 32687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:35581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46617/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:53.622547 32687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:53.623986 32687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:53.632799 32693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.632793 32699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.632789 32694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:53.633044 32687 server_base.cc:1034] running on GCE node
I20260806 02:50:53.633706 32687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:53.634847 32687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:53.636009 32687 hybrid_clock.cc:648] HybridClock initialized: now 1785984653635994 us; error 34 us; skew 500 ppm
I20260806 02:50:53.637832 32687 webserver.cc:460] Webserver started at http://127.24.130.190:37733/ using document root <none> and password file <none>
I20260806 02:50:53.638406 32687 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:53.638473 32687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:53.638787 32687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:53.640849 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "927d0568a690495399e16ae8d7319575"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "83327e66cefa143861ad1236ccb75c54"
server_key_iv: "22bb35b344b59cb54885271d963fe550"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.641433 32687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "927d0568a690495399e16ae8d7319575"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "83327e66cefa143861ad1236ccb75c54"
server_key_iv: "22bb35b344b59cb54885271d963fe550"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.645131 32687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:50:53.647516 32705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:53.648702 32687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:53.648849 32687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "927d0568a690495399e16ae8d7319575"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "83327e66cefa143861ad1236ccb75c54"
server_key_iv: "22bb35b344b59cb54885271d963fe550"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.648969 32687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:53.666101 32687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:53.666908 32687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:53.667137 32687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:53.674515 32687 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:35581
I20260806 02:50:53.674523 309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:35581 every 8 connection(s)
I20260806 02:50:53.675596 32687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:53.678884 310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:53.682680 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 32687
I20260806 02:50:53.682839 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:53.683183 25098 external_mini_cluster.cc:1428] Setting key a918544ce4d03e124b87381ce69d767e
I20260806 02:50:53.685192 310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575: Bootstrap starting.
I20260806 02:50:53.687749 310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:53.688611 310 log.cc:824] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:53.690699 310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575: No bootstrap required, opened a new log
I20260806 02:50:53.693318 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.685031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.24.130.190:35581)
0806 02:50:53.685746 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.685762 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.685798 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:50:53.686464 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.686481 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.686967 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.687249 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:53.687611 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.688671 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.689657 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.690156 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.690359 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.691296 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:53.691302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:53.691325 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:53.691747 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:53.691806 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:53.691826 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:53.691945 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:53.692062 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 02:50:53.692343 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":161,"threads_started":1}
I20260806 02:50:53.694051 310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927d0568a690495399e16ae8d7319575" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35581 } }
I20260806 02:50:53.694324 310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:53.694411 310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927d0568a690495399e16ae8d7319575, State: Initialized, Role: FOLLOWER
I20260806 02:50:53.694927 310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927d0568a690495399e16ae8d7319575" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35581 } }
I20260806 02:50:53.695060 310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:53.695149 310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:53.695289 310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:53.696338 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:53.696460 310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927d0568a690495399e16ae8d7319575" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35581 } }
I20260806 02:50:53.696817 310 leader_election.cc:302] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 927d0568a690495399e16ae8d7319575; no voters:
I20260806 02:50:53.697177 310 leader_election.cc:288] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:53.697268 316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:53.697475 316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [term 1 LEADER]: Becoming Leader. State: Replica: 927d0568a690495399e16ae8d7319575, State: Running, Role: LEADER
I20260806 02:50:53.697789 316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927d0568a690495399e16ae8d7319575" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35581 } }
I20260806 02:50:53.698522 310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:53.699115 319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927d0568a690495399e16ae8d7319575. Latest consensus state: current_term: 1 leader_uuid: "927d0568a690495399e16ae8d7319575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927d0568a690495399e16ae8d7319575" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35581 } } }
I20260806 02:50:53.699280 319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.699560 317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "927d0568a690495399e16ae8d7319575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927d0568a690495399e16ae8d7319575" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35581 } } }
I20260806 02:50:53.699671 317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.701264 325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:53.704177 325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:53.709844 325 catalog_manager.cc:1347] Generated new cluster ID: 93ea441749224742a9cda09240ec0441
I20260806 02:50:53.709928 325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:53.724483 325 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:53.725509 325 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:53.732496 325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 927d0568a690495399e16ae8d7319575: Generated new TSK 0
I20260806 02:50:53.733217 325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:53.749537 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.744672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.24.130.190:35581)
0806 02:50:53.744837 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.744842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.744966 (+ 124us) server_negotiation.cc:407] Connection header received
0806 02:50:53.745186 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.745188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.745237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.745356 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:50:53.745618 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.746176 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.747186 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.747420 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.747538 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.747877 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:50:53.749122 (+ 1245us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:50:53.749390 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:50:53.749630 314 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:50:53.749616 361 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:53.744551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:35581 (local address 127.0.0.1:52782)
0806 02:50:53.744868 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:53.744881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:53.745026 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:53.745356 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:53.745360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:53.745369 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:50:53.745502 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:53.745508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.746304 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.746308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:53.747045 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:53.747054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.747565 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.747569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:53.747656 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.747687 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:50:53.749289 (+ 1602us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:50:53.749442 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.130.190:35581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260806 02:50:53.750456 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 32687
E20260806 02:50:53.768208 362 webserver.cc:533] Webserver: sslize failed:
E20260806 02:50:53.768318 362 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T02:50:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:53Z Disabled control of system clock
I20260806 02:50:53.809371 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:34733
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:34733
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/testchainca.cert with env {}
W20260806 02:50:53.923159 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:53.923444 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:53.923484 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:53.927445 369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:53.927500 369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:53.927515 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:53.927532 369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:53.927548 369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:50:53.927564 369 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:50:53.927578 369 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:50:53.932928 369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:34733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:53.933987 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:53.935031 369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:53.943960 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.943948 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.943948 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:53.944380 369 server_base.cc:1034] running on GCE node
I20260806 02:50:53.945012 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:53.946188 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:53.947377 369 hybrid_clock.cc:648] HybridClock initialized: now 1785984653947347 us; error 42 us; skew 500 ppm
I20260806 02:50:53.949419 369 webserver.cc:460] Webserver started at http://127.24.130.190:44769/ using document root <none> and password file <none>
I20260806 02:50:53.950027 369 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:53.950095 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:53.950340 369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:53.952597 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "170b277b641d413d916c07e1ab40e86f"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "8eead96af6e1075a5829d44f080e6b59"
server_key_iv: "5ad378e394172a46ea98f101fa3fc21c"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.953197 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "170b277b641d413d916c07e1ab40e86f"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "8eead96af6e1075a5829d44f080e6b59"
server_key_iv: "5ad378e394172a46ea98f101fa3fc21c"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.957268 369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:50:53.960112 387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:53.961526 369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:53.961704 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "170b277b641d413d916c07e1ab40e86f"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-3kk6"
server_key: "8eead96af6e1075a5829d44f080e6b59"
server_key_iv: "5ad378e394172a46ea98f101fa3fc21c"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.961843 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:53.989265 369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:53.990106 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:53.990345 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:53.997978 369 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:34733
I20260806 02:50:53.997982 466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:34733 every 8 connection(s)
I20260806 02:50:53.999099 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:54.002019 467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:54.006309 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 369
I20260806 02:50:54.006454 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:54.006771 25098 external_mini_cluster.cc:1428] Setting key a4c0f340dccb2d707203fe6522244173
I20260806 02:50:54.008311 467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f: Bootstrap starting.
I20260806 02:50:54.010583 467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:54.011399 467 log.cc:824] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:54.013388 467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f: No bootstrap required, opened a new log
I20260806 02:50:54.016505 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:54.008274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.24.130.190:34733)
0806 02:50:54.008945 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:54.008961 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:54.008997 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:50:54.009619 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:54.009636 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:54.010037 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:54.010386 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:54.010891 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:54.011959 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:54.013047 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:54.013574 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:54.013753 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:54.014699 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:54.014705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:54.014729 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:54.015139 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:54.015196 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:54.015216 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:54.015334 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:54.015495 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 02:50:54.015765 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
I20260806 02:50:54.017339 467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170b277b641d413d916c07e1ab40e86f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34733 } }
I20260806 02:50:54.017633 467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:54.017719 467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170b277b641d413d916c07e1ab40e86f, State: Initialized, Role: FOLLOWER
I20260806 02:50:54.018168 467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170b277b641d413d916c07e1ab40e86f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34733 } }
I20260806 02:50:54.018299 467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:54.018386 467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:54.018505 467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:54.019642 467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170b277b641d413d916c07e1ab40e86f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34733 } }
I20260806 02:50:54.019876 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:54.020018 467 leader_election.cc:302] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 170b277b641d413d916c07e1ab40e86f; no voters:
I20260806 02:50:54.020373 467 leader_election.cc:288] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:54.020479 474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:54.020768 474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [term 1 LEADER]: Becoming Leader. State: Replica: 170b277b641d413d916c07e1ab40e86f, State: Running, Role: LEADER
I20260806 02:50:54.021127 474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170b277b641d413d916c07e1ab40e86f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34733 } }
I20260806 02:50:54.021703 467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:54.022374 477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170b277b641d413d916c07e1ab40e86f. Latest consensus state: current_term: 1 leader_uuid: "170b277b641d413d916c07e1ab40e86f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170b277b641d413d916c07e1ab40e86f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34733 } } }
I20260806 02:50:54.022529 477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:54.022889 476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "170b277b641d413d916c07e1ab40e86f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "170b277b641d413d916c07e1ab40e86f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34733 } } }
I20260806 02:50:54.022992 476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:54.023252 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:54.026494 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:54.032807 486 catalog_manager.cc:1347] Generated new cluster ID: 6a485bd1f850494db0d680dda69427ac
I20260806 02:50:54.032924 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:54.049962 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:54.051023 486 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:54.060086 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 170b277b641d413d916c07e1ab40e86f: Generated new TSK 0
I20260806 02:50:54.061029 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:54.138011 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:54.076031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.24.130.190:34733)
0806 02:50:54.076172 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:54.076177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:54.076380 (+ 203us) server_negotiation.cc:407] Connection header received
0806 02:50:54.076600 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:54.076603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:54.076655 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:54.076782 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:50:54.077045 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:54.077605 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:54.078512 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:54.078810 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:54.078928 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:54.079278 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:50:54.137593 (+ 58315us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:50:54.137884 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42952: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:50:54.138128 470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42952: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:50:54.138340 509 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:54.075906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:34733 (local address 127.0.0.1:42952)
0806 02:50:54.076276 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:54.076290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:54.076443 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:54.076781 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:54.076784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:54.076792 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:50:54.076928 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:54.076935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:54.077731 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:54.077735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:54.078381 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:54.078389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:54.078945 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:54.078949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:54.079051 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:54.079077 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:50:54.137874 (+ 58797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:50:54.138115 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.130.190:34733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260806 02:50:54.139407 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 369
2026-08-06T02:50:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (370 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (131 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (130 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:54 dist-test-slave-3kk6 krb5kdc[526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:54 dist-test-slave-3kk6 krb5kdc[526](info): set up 2 sockets
Aug 06 02:50:54 dist-test-slave-3kk6 krb5kdc[526](info): commencing operation
krb5kdc: starting...
W20260806 02:50:55.627569 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:55 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:50:55.762297 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:46695
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:46695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:55.881613 545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:55.881971 545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:55.882053 545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:55.886210 545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:55.886332 545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:55.886370 545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:55.886412 545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:55.886445 545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:55.892136 545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:46695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:46695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:55.893303 545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:55.894407 545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:55.903214 551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:55.903388 545 server_base.cc:1034] running on GCE node
W20260806 02:50:55.903245 554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:55.903214 552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:55.904237 545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:55.905352 545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:55.906515 545 hybrid_clock.cc:648] HybridClock initialized: now 1785984655906483 us; error 50 us; skew 500 ppm
Aug 06 02:50:55 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984655, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:55.912268 545 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:50:55.913220 545 webserver.cc:460] Webserver started at http://127.24.130.190:44993/ using document root <none> and password file <none>
I20260806 02:50:55.913803 545 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:55.913910 545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:55.914162 545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:55.916463 545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "d828b7c1d63441488e7834e1b77f2fd6"
format_stamp: "Formatted at 2026-08-06 02:50:55 on dist-test-slave-3kk6"
server_key: "7124122c5e68d72e8057ed304acc2442"
server_key_iv: "c6261b99f39341e09d3bb172c2ec3177"
server_key_version: "encryptionkey@0"
I20260806 02:50:55.917073 545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "d828b7c1d63441488e7834e1b77f2fd6"
format_stamp: "Formatted at 2026-08-06 02:50:55 on dist-test-slave-3kk6"
server_key: "7124122c5e68d72e8057ed304acc2442"
server_key_iv: "c6261b99f39341e09d3bb172c2ec3177"
server_key_version: "encryptionkey@0"
I20260806 02:50:55.921130 545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:50:55.923609 564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:55.924849 545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:55.925001 545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "d828b7c1d63441488e7834e1b77f2fd6"
format_stamp: "Formatted at 2026-08-06 02:50:55 on dist-test-slave-3kk6"
server_key: "7124122c5e68d72e8057ed304acc2442"
server_key_iv: "c6261b99f39341e09d3bb172c2ec3177"
server_key_version: "encryptionkey@0"
I20260806 02:50:55.925124 545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:55.945710 545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:55.949102 545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:55.949353 545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:55.957122 545 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:46695
I20260806 02:50:55.957137 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:46695 every 8 connection(s)
I20260806 02:50:55.958190 545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:55.960605 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 545
I20260806 02:50:55.960752 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:55.961092 25098 external_mini_cluster.cc:1428] Setting key 5b0e38067442fd04aa7dc71a60e60e68
I20260806 02:50:55.961565 637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:55.968053 637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6: Bootstrap starting.
Aug 06 02:50:55 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:55.970837 637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:55.971818 637 log.cc:824] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:55.974375 637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6: No bootstrap required, opened a new log
I20260806 02:50:55.976725 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:55.962724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.24.130.190:46695)
0806 02:50:55.963212 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:55.963227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:55.963266 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:50:55.963942 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:55.963958 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:55.964300 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:55.964651 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:55.965532 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:55.966569 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:55.967482 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:55.967966 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:55.968145 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:55.971063 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:55.971072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:55.971094 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:55.971134 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:55.973120 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:55.973573 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:55.973580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:55.973586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:55.973729 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:55.974007 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:55.974010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:55.974012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:55.974558 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:55.974794 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:55.975135 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 02:50:55.975462 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":166,"threads_started":1}
I20260806 02:50:55.978554 637 raft_consensus.cc:348] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d828b7c1d63441488e7834e1b77f2fd6" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46695 } }
I20260806 02:50:55.978863 637 raft_consensus.cc:374] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:55.978977 637 raft_consensus.cc:729] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d828b7c1d63441488e7834e1b77f2fd6, State: Initialized, Role: FOLLOWER
I20260806 02:50:55.979494 637 consensus_queue.cc:260] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d828b7c1d63441488e7834e1b77f2fd6" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46695 } }
I20260806 02:50:55.979656 637 raft_consensus.cc:388] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:55.979727 637 raft_consensus.cc:482] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:55.979830 637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:55.980875 637 raft_consensus.cc:504] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d828b7c1d63441488e7834e1b77f2fd6" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46695 } }
I20260806 02:50:55.981236 637 leader_election.cc:302] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d828b7c1d63441488e7834e1b77f2fd6; no voters:
I20260806 02:50:55.981577 637 leader_election.cc:288] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:55.981693 643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:55.981950 643 raft_consensus.cc:686] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [term 1 LEADER]: Becoming Leader. State: Replica: d828b7c1d63441488e7834e1b77f2fd6, State: Running, Role: LEADER
I20260806 02:50:55.982357 643 consensus_queue.cc:237] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d828b7c1d63441488e7834e1b77f2fd6" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46695 } }
I20260806 02:50:55.982903 637 sys_catalog.cc:564] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:55.984241 647 sys_catalog.cc:455] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d828b7c1d63441488e7834e1b77f2fd6. Latest consensus state: current_term: 1 leader_uuid: "d828b7c1d63441488e7834e1b77f2fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d828b7c1d63441488e7834e1b77f2fd6" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46695 } } }
I20260806 02:50:55.984268 644 sys_catalog.cc:455] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d828b7c1d63441488e7834e1b77f2fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d828b7c1d63441488e7834e1b77f2fd6" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46695 } } }
I20260806 02:50:55.984372 647 sys_catalog.cc:458] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:55.984390 644 sys_catalog.cc:458] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:55.984748 657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:55.988025 657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:55.994076 657 catalog_manager.cc:1347] Generated new cluster ID: 5a1caa219d5843aa882206466062b29c
I20260806 02:50:55.994167 657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:56.009963 657 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:56.011408 657 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:56.018993 657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d828b7c1d63441488e7834e1b77f2fd6: Generated new TSK 0
I20260806 02:50:56.019893 657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:50:56.057888 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46695
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:56.179337 675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:56.179679 675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:56.179759 675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:56.183947 675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:56.184038 675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:56.184128 675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:56.189709 675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:56.190960 675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:56.192052 675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:56.202744 681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:56.202744 684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:56.202898 675 server_base.cc:1034] running on GCE node
W20260806 02:50:56.202786 682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:56.203508 675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:56.204172 675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:56.205351 675 hybrid_clock.cc:648] HybridClock initialized: now 1785984656205330 us; error 30 us; skew 500 ppm
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.208310 675 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:50:56.209213 675 webserver.cc:460] Webserver started at http://127.24.130.129:39759/ using document root <none> and password file <none>
I20260806 02:50:56.209766 675 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:56.209826 675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:56.210000 675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:56.212206 675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "649026f34be54d7d962a32f63f2e3057"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "3bde36659fb24b6140f56b00f2de5288"
server_key_iv: "d71e1e182263675449cc30b6eb64bf83"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.212766 675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "649026f34be54d7d962a32f63f2e3057"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "3bde36659fb24b6140f56b00f2de5288"
server_key_iv: "d71e1e182263675449cc30b6eb64bf83"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.216815 675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:50:56.219507 694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.220815 675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:50:56.220980 675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "649026f34be54d7d962a32f63f2e3057"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "3bde36659fb24b6140f56b00f2de5288"
server_key_iv: "d71e1e182263675449cc30b6eb64bf83"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.221117 675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:56.235024 675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:56.238197 675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:56.238435 675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:56.239142 675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:56.240013 675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:56.240088 675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.240162 675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:56.240206 675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.249881 675 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:35557
I20260806 02:50:56.249912 857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:35557 every 8 connection(s)
I20260806 02:50:56.250885 675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:56.256266 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 675
I20260806 02:50:56.256381 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:56.256691 25098 external_mini_cluster.cc:1428] Setting key 11f41c4fb598614b6adf412ad8f478a2
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:56.266875 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.252816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:52215 (local address 127.24.130.190:46695)
0806 02:50:56.252970 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.252974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.253815 (+ 841us) server_negotiation.cc:407] Connection header received
0806 02:50:56.255357 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.255361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.255424 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.255509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.257228 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.258008 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.259323 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.259573 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.259693 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.262479 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.262486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.262492 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:56.262528 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.264454 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.264977 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.264985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.264987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.265098 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.265462 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.265467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.265471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.265776 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.265895 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.266456 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.266611 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:50:56.267704 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.253204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:46695 (local address 127.24.130.129:52215)
0806 02:50:56.253661 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:56.253706 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:56.255129 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:56.255686 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:56.255692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:56.256064 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:56.256665 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:56.256681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.258170 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.258175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:56.259189 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:56.259199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.259737 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.259741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:56.259929 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.260596 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:56.260613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:56.262274 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:56.264578 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:56.264583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:56.264597 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:56.264863 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:56.265212 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:56.265214 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:56.265216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:56.265361 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:56.265894 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:56.265900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:56.266143 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 02:50:56.266330 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 02:50:56.266746 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:50:56.269115 858 heartbeater.cc:347] Connected to a master server at 127.24.130.190:46695
I20260806 02:50:56.269405 858 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:56.269970 858 heartbeater.cc:511] Master 127.24.130.190:46695 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:56.271631 588 ts_manager.cc:194] Registered new tserver with Master: 649026f34be54d7d962a32f63f2e3057 (127.24.130.129:35557)
I20260806 02:50:56.272768 588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:52215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:50:56.288251 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46695
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:56.408569 865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:56.408871 865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:56.408915 865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:56.413074 865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:56.413158 865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:56.413226 865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:56.418717 865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:56.419842 865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:56.420869 865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:56.431243 872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:56.431257 877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:56.431272 873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:56.431628 865 server_base.cc:1034] running on GCE node
I20260806 02:50:56.432078 865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:56.432787 865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:56.433988 865 hybrid_clock.cc:648] HybridClock initialized: now 1785984656433974 us; error 31 us; skew 500 ppm
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.437104 865 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:50:56.438042 865 webserver.cc:460] Webserver started at http://127.24.130.130:32835/ using document root <none> and password file <none>
I20260806 02:50:56.438637 865 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:56.438791 865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:56.439049 865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:56.441193 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "97be989854c947e08305bc2022e9305a"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "ad4c1e8f411c59a665fdacdfc01c0737"
server_key_iv: "81d5be6d86f9cb88204c668b18695721"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.441794 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "97be989854c947e08305bc2022e9305a"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "ad4c1e8f411c59a665fdacdfc01c0737"
server_key_iv: "81d5be6d86f9cb88204c668b18695721"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.445667 865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 02:50:56.448256 884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.449415 865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:56.449553 865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "97be989854c947e08305bc2022e9305a"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "ad4c1e8f411c59a665fdacdfc01c0737"
server_key_iv: "81d5be6d86f9cb88204c668b18695721"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.449674 865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:56.485154 865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:56.488957 865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:56.489190 865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:56.489848 865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:56.490808 865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:56.490886 865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.490962 865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:56.491016 865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.500451 865 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:34353
I20260806 02:50:56.500473 1046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:34353 every 8 connection(s)
I20260806 02:50:56.501461 865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:56.507177 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 865
I20260806 02:50:56.507341 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:56.507700 25098 external_mini_cluster.cc:1428] Setting key 876634a56b36738c4fd786f5ea362d1d
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:56.517023 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.503506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:59475 (local address 127.24.130.190:46695)
0806 02:50:56.503688 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.503694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.504587 (+ 893us) server_negotiation.cc:407] Connection header received
0806 02:50:56.505988 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.505991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.506043 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.506121 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.507547 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.508294 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.509638 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.510037 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.510175 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.513077 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.513081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.513083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:56.513109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.514791 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.515355 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.515362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.515366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.515456 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.515821 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.515824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.515826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.516038 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.516151 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.516688 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.516843 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:50:56.517874 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.503921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:46695 (local address 127.24.130.130:59475)
0806 02:50:56.504416 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:56.504464 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:56.505775 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:56.506301 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:56.506307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:56.506778 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:56.507361 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:56.507383 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.508438 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.508442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:56.509509 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:56.509520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.510203 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.510205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:56.510356 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.511075 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:56.511093 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:56.512889 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:56.514926 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:56.514932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:56.514950 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:56.515242 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:56.515556 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:56.515559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:56.515561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:56.515716 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:56.516151 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:56.516160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:56.516442 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 02:50:56.516647 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 02:50:56.517003 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:50:56.519464 1047 heartbeater.cc:347] Connected to a master server at 127.24.130.190:46695
I20260806 02:50:56.519728 1047 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:56.520277 1047 heartbeater.cc:511] Master 127.24.130.190:46695 requested a full tablet report, sending...
I20260806 02:50:56.521442 588 ts_manager.cc:194] Registered new tserver with Master: 97be989854c947e08305bc2022e9305a (127.24.130.130:34353)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
I20260806 02:50:56.522343 588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:59475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:50:56.539685 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46695
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:56.659605 1057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:56.659916 1057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:56.659962 1057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:56.664130 1057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:56.664219 1057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:56.664281 1057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:56.669756 1057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:56.670997 1057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:56.672462 1057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:56.682693 1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:56.682711 1069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:56.682768 1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:56.683163 1057 server_base.cc:1034] running on GCE node
I20260806 02:50:56.683626 1057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:56.684367 1057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:56.685544 1057 hybrid_clock.cc:648] HybridClock initialized: now 1785984656685509 us; error 61 us; skew 500 ppm
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.688650 1057 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:50:56.689720 1057 webserver.cc:460] Webserver started at http://127.24.130.131:34089/ using document root <none> and password file <none>
I20260806 02:50:56.690366 1057 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:56.690436 1057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:56.690725 1057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:56.693068 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "ae06ad1f56a440e7aa1511eb5fd75626"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "c7b5ed070ac76aac2955cd917722445a"
server_key_iv: "a67c240e65d1fce1389bea0ac3bdab43"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.693735 1057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "ae06ad1f56a440e7aa1511eb5fd75626"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "c7b5ed070ac76aac2955cd917722445a"
server_key_iv: "a67c240e65d1fce1389bea0ac3bdab43"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.698045 1057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 02:50:56.700850 1080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.702260 1057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:56.702401 1057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "ae06ad1f56a440e7aa1511eb5fd75626"
format_stamp: "Formatted at 2026-08-06 02:50:56 on dist-test-slave-3kk6"
server_key: "c7b5ed070ac76aac2955cd917722445a"
server_key_iv: "a67c240e65d1fce1389bea0ac3bdab43"
server_key_version: "encryptionkey@0"
I20260806 02:50:56.702544 1057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:56.728470 1057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:56.731948 1057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:56.732191 1057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:56.732803 1057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:56.733650 1057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:56.733702 1057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.733744 1057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:56.733767 1057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:56.743582 1057 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:46815
I20260806 02:50:56.743666 1251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:46815 every 8 connection(s)
I20260806 02:50:56.744602 1057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:56.748476 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 1057
I20260806 02:50:56.748620 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:56.748943 25098 external_mini_cluster.cc:1428] Setting key ed9fc72d20ed4086037fe7bb5d086e70
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:50:56.760411 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.746535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:46737 (local address 127.24.130.190:46695)
0806 02:50:56.746744 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.746750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.747673 (+ 923us) server_negotiation.cc:407] Connection header received
0806 02:50:56.749254 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.749260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.749335 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.749416 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.751199 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.751852 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.753323 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.753600 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.753711 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.756418 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.756422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.756424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:56.756451 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.758046 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.758684 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.758688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.758690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.758783 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.759179 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.759182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.759184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.759399 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.759513 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.760070 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.760228 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:50:56.761279 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.746967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:46695 (local address 127.24.130.131:46737)
0806 02:50:56.747500 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:56.747548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:56.748978 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:56.749644 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:56.749651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:56.750254 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:56.751035 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:56.751058 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.752071 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.752075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:56.753189 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:56.753200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.753734 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.753737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:56.753888 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.754524 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:56.754542 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:56.756220 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:56.758191 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:56.758198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:56.758216 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:56.758515 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:56.758889 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:56.758892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:56.758894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:56.759072 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:56.759512 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:56.759520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:56.759810 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 02:50:56.760012 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 02:50:56.760383 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":131,"threads_started":1}
I20260806 02:50:56.762575 1252 heartbeater.cc:347] Connected to a master server at 127.24.130.190:46695
I20260806 02:50:56.762895 1252 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:56.763492 1252 heartbeater.cc:511] Master 127.24.130.190:46695 requested a full tablet report, sending...
I20260806 02:50:56.764524 588 ts_manager.cc:194] Registered new tserver with Master: ae06ad1f56a440e7aa1511eb5fd75626 (127.24.130.131:46815)
I20260806 02:50:56.765133 588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:46737
I20260806 02:50:56.775727 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/test-admin.keytab.
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.795681 25098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:50:56.799443 1266 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:56.797734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.24.130.129:35557)
0806 02:50:56.798088 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.798099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.798133 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:50:56.798204 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.798211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.798498 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.798639 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.799249 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58466: BlockingRecv error: recv got EOF from 127.0.0.1:58466 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":126,"threads_started":1}
W20260806 02:50:56.799546 25098 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.800881 25098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.802024 25098 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:56.812603 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.803140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.24.130.129:35557)
0806 02:50:56.803269 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.803273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.803500 (+ 227us) server_negotiation.cc:407] Connection header received
0806 02:50:56.803600 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.803602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.803644 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.803720 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.804307 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.805106 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.805929 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.806317 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.806426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.809097 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.809109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.809125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:56.809154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.810850 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.811284 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.811289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.811290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.811374 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.811614 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.811616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.811617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.811905 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.812086 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.812230 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.812407 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:50:56.816623 1266 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:56.815671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.24.130.129:35557)
0806 02:50:56.815842 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.815845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.815859 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:56.815980 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.815982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.816020 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.816087 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.816493 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58486: BlockingRecv error: recv got EOF from 127.0.0.1:58486 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 02:50:56.816949 25098 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc)
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.818101 25098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.819115 25098 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:56.827968 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.820132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.24.130.129:35557)
0806 02:50:56.820273 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.820276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.820292 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:50:56.820418 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.820420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.820460 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.820526 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.820978 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.821477 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.822218 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.822442 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.822515 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.824817 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.824821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.824822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:56.824848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.826342 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.826781 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.826784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.826785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.826865 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.827102 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.827104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.827105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.827291 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.827423 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.827650 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.827770 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:50:56.830965 1266 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:56.830012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.24.130.129:35557)
0806 02:50:56.830168 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.830172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.830242 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:50:56.830340 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.830342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.830378 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.830437 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.830854 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58500: BlockingRecv error: recv got EOF from 127.0.0.1:58500 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:50:56.831063 25098 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984624291587-25098-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.832196 25098 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:56.833134 25098 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:50:56.842113 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.834061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.24.130.129:35557)
0806 02:50:56.834197 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.834202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.834326 (+ 124us) server_negotiation.cc:407] Connection header received
0806 02:50:56.834496 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.834504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.834543 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.834623 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.835183 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.835673 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.836460 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:56.836673 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:56.836777 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:56.839207 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.839210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.839212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:56.839236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.840703 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.841094 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.841097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.841098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.841171 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:56.841402 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:56.841404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:56.841405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:56.841567 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.841680 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.841901 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.841993 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:50:56.843318 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 675
I20260806 02:50:56.858260 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 865
I20260806 02:50:56.870611 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 1057
I20260806 02:50:56.887234 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 545
2026-08-06T02:50:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2486 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[1284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[1284](info): set up 2 sockets
Aug 06 02:50:56 dist-test-slave-3kk6 krb5kdc[1284](info): commencing operation
krb5kdc: starting...
W20260806 02:50:58.113978 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:50:58 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab.
Entry for principal oryx/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab.
I20260806 02:50:58.254608 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44675
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab
--principal=oryx/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:58.381680 1303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:58.382030 1303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:58.382076 1303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:58.386639 1303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:58.386773 1303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:58.386802 1303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:58.386821 1303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:58.386838 1303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:58.392779 1303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab
--principal=oryx/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:58.394083 1303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:58.395296 1303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:58.405104 1303 server_base.cc:1034] running on GCE node
W20260806 02:50:58.405015 1310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:58.405015 1308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:58.405015 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:58.406050 1303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:58.407393 1303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:58.408595 1303 hybrid_clock.cc:648] HybridClock initialized: now 1785984658408569 us; error 44 us; skew 500 ppm
Aug 06 02:50:58 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:58.411712 1303 init.cc:377] Logged in from keytab as oryx/127.24.130.190@KRBTEST.COM (short username oryx)
I20260806 02:50:58.412747 1303 webserver.cc:460] Webserver started at http://127.24.130.190:35167/ using document root <none> and password file <none>
I20260806 02:50:58.413370 1303 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:58.413470 1303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:58.413730 1303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:58.416147 1303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "f26b4ce127e949b59618b96a68c890de"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "d87cf511132bb4ca769dad0afae24ac6"
server_key_iv: "b41e7cdae19d4739c73e817d4583dae7"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.416839 1303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "f26b4ce127e949b59618b96a68c890de"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "d87cf511132bb4ca769dad0afae24ac6"
server_key_iv: "b41e7cdae19d4739c73e817d4583dae7"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.421162 1303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:58.424182 1327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.425522 1303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:58.425680 1303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "f26b4ce127e949b59618b96a68c890de"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "d87cf511132bb4ca769dad0afae24ac6"
server_key_iv: "b41e7cdae19d4739c73e817d4583dae7"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.425817 1303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:58.437973 1303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:58.441428 1303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:58.441670 1303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:58.449473 1303 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:44675
I20260806 02:50:58.449792 1398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:44675 every 8 connection(s)
I20260806 02:50:58.450635 1303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:58.453781 1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:58.453992 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 1303
I20260806 02:50:58.454154 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:50:58.454485 25098 external_mini_cluster.cc:1428] Setting key f256df3b39019ee05cb78720d0c860ec
I20260806 02:50:58.461153 1399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Bootstrap starting.
I20260806 02:50:58.463722 1399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Neither blocks nor log segments found. Creating new log.
Aug 06 02:50:58 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:50:58.464612 1399 log.cc:824] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:58.466924 1399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: No bootstrap required, opened a new log
I20260806 02:50:58.469919 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:58.456130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.24.130.190:44675)
0806 02:50:58.456696 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:58.456713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:58.456785 (+ 72us) server_negotiation.cc:407] Connection header received
0806 02:50:58.457504 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:58.457524 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:58.457894 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:58.458287 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:58.459244 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.460317 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.461217 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.461794 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.461978 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:58.464829 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:58.464835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:58.464850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:58.464881 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:58.466868 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.467398 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.467404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.467409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.467530 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.467850 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.467858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.467860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.468284 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:58.468510 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:58.468818 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 02:50:58.469140 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":180,"threads_started":1}
I20260806 02:50:58.471024 1399 raft_consensus.cc:348] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } }
I20260806 02:50:58.471331 1399 raft_consensus.cc:374] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:58.471418 1399 raft_consensus.cc:729] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26b4ce127e949b59618b96a68c890de, State: Initialized, Role: FOLLOWER
I20260806 02:50:58.471935 1399 consensus_queue.cc:260] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } }
I20260806 02:50:58.472097 1399 raft_consensus.cc:388] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:58.472174 1399 raft_consensus.cc:482] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:58.472290 1399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:58.473361 1399 raft_consensus.cc:504] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } }
I20260806 02:50:58.473732 1399 leader_election.cc:302] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f26b4ce127e949b59618b96a68c890de; no voters:
I20260806 02:50:58.474155 1399 leader_election.cc:288] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:58.474289 1407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:58.474570 1407 raft_consensus.cc:686] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 LEADER]: Becoming Leader. State: Replica: f26b4ce127e949b59618b96a68c890de, State: Running, Role: LEADER
I20260806 02:50:58.475032 1407 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } }
I20260806 02:50:58.475576 1399 sys_catalog.cc:564] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:58.476933 1408 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } }
I20260806 02:50:58.477095 1408 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:58.477068 1409 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26b4ce127e949b59618b96a68c890de. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } }
I20260806 02:50:58.477275 1409 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:58.480661 1428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:58.480753 1428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:58.480854 1422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:58.481599 1422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:58.488418 1422 catalog_manager.cc:1347] Generated new cluster ID: 203476151937425f84de8a39b4accf79
I20260806 02:50:58.488528 1422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:58.499907 1422 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:58.500955 1422 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:58.506556 1422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Generated new TSK 0
I20260806 02:50:58.507505 1422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab.
Entry for principal oryx/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab.
I20260806 02:50:58.550275 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44675
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:58.677830 1435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:58.678193 1435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:58.678282 1435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:58.682551 1435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:58.682721 1435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:58.682829 1435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:50:58.688717 1435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:58.690033 1435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:58.691210 1435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:58.701880 1446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:58.701882 1443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:58.701915 1444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:58.702162 1435 server_base.cc:1034] running on GCE node
I20260806 02:50:58.702807 1435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:58.703554 1435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:58.704763 1435 hybrid_clock.cc:648] HybridClock initialized: now 1785984658704745 us; error 33 us; skew 500 ppm
Aug 06 02:50:58 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:58.707978 1435 init.cc:377] Logged in from keytab as oryx/127.24.130.129@KRBTEST.COM (short username oryx)
I20260806 02:50:58.709017 1435 webserver.cc:460] Webserver started at http://127.24.130.129:37867/ using document root <none> and password file <none>
I20260806 02:50:58.709589 1435 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:58.709653 1435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:58.709831 1435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:58.712096 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "3503ae3c442040b4af4744a432245e77"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "43fdf3e5f98a1a44b4e329318116a09b"
server_key_iv: "2aa47e62383f6cd83401a7ae73f2178b"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.712670 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "3503ae3c442040b4af4744a432245e77"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "43fdf3e5f98a1a44b4e329318116a09b"
server_key_iv: "2aa47e62383f6cd83401a7ae73f2178b"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.717298 1435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:58.720026 1463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.721644 1435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:58.721823 1435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "3503ae3c442040b4af4744a432245e77"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "43fdf3e5f98a1a44b4e329318116a09b"
server_key_iv: "2aa47e62383f6cd83401a7ae73f2178b"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.721962 1435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:58.758637 1435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:58.762735 1435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:58.762995 1435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:58.763720 1435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:58.764685 1435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:58.764810 1435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.764891 1435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:58.764937 1435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.774818 1435 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:42361
I20260806 02:50:58.774840 1612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:42361 every 8 connection(s)
I20260806 02:50:58.775980 1435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:58.780375 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 1435
I20260806 02:50:58.780498 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:58.780835 25098 external_mini_cluster.cc:1428] Setting key 69d7d9cfd3a0306e9ec9031bab3c8ab1
Aug 06 02:50:58 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:50:58.792687 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:58.778196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:57121 (local address 127.24.130.190:44675)
0806 02:50:58.778372 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:58.778375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:58.779345 (+ 970us) server_negotiation.cc:407] Connection header received
0806 02:50:58.780868 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:58.780874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:58.780942 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:58.781043 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:58.782800 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.783601 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.784897 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.785145 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.785270 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:58.788510 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:58.788513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:58.788515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:58.788543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:58.790174 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.790887 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.790890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.790891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.790979 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.791387 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.791390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.791392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.791613 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:58.791750 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:58.792343 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 02:50:58.792504 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:50:58.793740 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:58.778580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:57121)
0806 02:50:58.779160 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:58.779210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:58.780627 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:58.781265 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:58.781277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:58.781830 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:58.782599 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:58.782618 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.783739 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.783742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:58.784747 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:58.784757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.785322 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.785326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:58.785541 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:58.786285 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:58.786303 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:58.788283 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:58.790328 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:58.790335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:58.790364 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:58.790747 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:58.791096 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:58.791099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:58.791101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:58.791271 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:58.791751 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:58.791760 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:58.792070 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 02:50:58.792280 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 02:50:58.792699 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":126,"threads_started":1}
I20260806 02:50:58.795222 1617 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.130@KRBTEST.COM" created.
I20260806 02:50:58.795514 1617 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:58.796125 1617 heartbeater.cc:511] Master 127.24.130.190:44675 requested a full tablet report, sending...
I20260806 02:50:58.797940 1352 ts_manager.cc:194] Registered new tserver with Master: 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361)
I20260806 02:50:58.799263 1352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:57121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab.
Entry for principal oryx/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab.
I20260806 02:50:58.814071 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44675
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:58.935911 1626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:58.936233 1626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:58.936280 1626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:58.941133 1626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:58.941221 1626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:58.941286 1626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:50:58.947286 1626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:58.948443 1626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:58.949492 1626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:58.960058 1632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:58.960284 1626 server_base.cc:1034] running on GCE node
W20260806 02:50:58.960071 1639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:58.960071 1634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:58.960902 1626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:58.961645 1626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:58.962833 1626 hybrid_clock.cc:648] HybridClock initialized: now 1785984658962799 us; error 54 us; skew 500 ppm
Aug 06 02:50:58 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:58.966171 1626 init.cc:377] Logged in from keytab as oryx/127.24.130.130@KRBTEST.COM (short username oryx)
I20260806 02:50:58.967272 1626 webserver.cc:460] Webserver started at http://127.24.130.130:38627/ using document root <none> and password file <none>
I20260806 02:50:58.967841 1626 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:58.967906 1626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:58.968082 1626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:58.970428 1626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "30a374a29e454d9f9ef138d36e35e151"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "30c762a443b71c27b0f89407d5845678"
server_key_iv: "bfcf784fb737acea69844c9655bf5015"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.971072 1626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "30a374a29e454d9f9ef138d36e35e151"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "30c762a443b71c27b0f89407d5845678"
server_key_iv: "bfcf784fb737acea69844c9655bf5015"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.975186 1626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:58.978039 1651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.979463 1626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:58.979600 1626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "30a374a29e454d9f9ef138d36e35e151"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "30c762a443b71c27b0f89407d5845678"
server_key_iv: "bfcf784fb737acea69844c9655bf5015"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.979724 1626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:58.993873 1626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:58.997576 1626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:58.997817 1626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:58.998517 1626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:58.999485 1626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:58.999564 1626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.999641 1626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:58.999693 1626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.010453 1626 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:35157
I20260806 02:50:59.010475 1808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:35157 every 8 connection(s)
I20260806 02:50:59.011554 1626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:59.013309 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 1626
I20260806 02:50:59.013458 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:59.013795 25098 external_mini_cluster.cc:1428] Setting key 1aed488e699d360d9ad2be2dffae7c52
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:50:59.028463 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.013714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:56375 (local address 127.24.130.190:44675)
0806 02:50:59.013862 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.013868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.014921 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 02:50:59.016970 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.016972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.017017 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.017089 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.018613 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.019457 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.020681 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.020925 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.021046 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.024097 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.024103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.024106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.024138 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.026050 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.026562 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.026565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.026567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.026701 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.027145 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.027151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.027155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.027393 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.027477 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.028034 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.028178 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.131@KRBTEST.COM" created.
I20260806 02:50:59.029134 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.014222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:56375)
0806 02:50:59.014749 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.014810 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.016725 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.017323 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.017335 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.017908 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.018453 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.018469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.019601 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.019604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.020526 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.020536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.021095 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.021099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.021288 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.021967 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.021985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.023890 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.026171 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.026176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.026190 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.026463 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.026795 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.026797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.026798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.027009 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.027494 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.027499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.027758 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.027937 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.028226 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":149,"threads_started":1}
I20260806 02:50:59.030475 1809 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44675
I20260806 02:50:59.030774 1809 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:59.031405 1809 heartbeater.cc:511] Master 127.24.130.190:44675 requested a full tablet report, sending...
I20260806 02:50:59.032657 1352 ts_manager.cc:194] Registered new tserver with Master: 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157)
I20260806 02:50:59.033320 1352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:56375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab.
Entry for principal oryx/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab.
I20260806 02:50:59.049638 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44675
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:59.186506 1817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:59.186909 1817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:59.187003 1817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:59.191172 1817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:59.191288 1817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:59.191395 1817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:50:59.197137 1817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:59.198455 1817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:59.199657 1817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:59.210784 1817 server_base.cc:1034] running on GCE node
W20260806 02:50:59.210819 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.210707 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.210718 1828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:59.211412 1817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:59.212154 1817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:59.213330 1817 hybrid_clock.cc:648] HybridClock initialized: now 1785984659213315 us; error 31 us; skew 500 ppm
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:59.216452 1817 init.cc:377] Logged in from keytab as oryx/127.24.130.131@KRBTEST.COM (short username oryx)
I20260806 02:50:59.217458 1817 webserver.cc:460] Webserver started at http://127.24.130.131:44793/ using document root <none> and password file <none>
I20260806 02:50:59.218024 1817 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:59.218086 1817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:59.218266 1817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:59.220714 1817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "17b9266ccbe346fc9d20626d3129cbfa"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "f6c952ece7e9e9bd8abe367d9bdd24e2"
server_key_iv: "8b16ea19ac60a852514653bb6649161a"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.221372 1817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "17b9266ccbe346fc9d20626d3129cbfa"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "f6c952ece7e9e9bd8abe367d9bdd24e2"
server_key_iv: "8b16ea19ac60a852514653bb6649161a"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.225756 1817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:59.228649 1838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.230198 1817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:59.230336 1817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "17b9266ccbe346fc9d20626d3129cbfa"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "f6c952ece7e9e9bd8abe367d9bdd24e2"
server_key_iv: "8b16ea19ac60a852514653bb6649161a"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.230468 1817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:59.263983 1817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:59.267489 1817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:59.267738 1817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:59.268416 1817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:59.269323 1817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:59.269373 1817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.269444 1817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:59.269485 1817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.279594 1817 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:33791
I20260806 02:50:59.279618 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:33791 every 8 connection(s)
I20260806 02:50:59.280630 1817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:59.281026 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 1817
I20260806 02:50:59.281180 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:59.281541 25098 external_mini_cluster.cc:1428] Setting key dce378c6cdc3c397a0941c57b1f70ec8
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:50:59.296686 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.282707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:35649 (local address 127.24.130.190:44675)
0806 02:50:59.282854 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.282859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.284003 (+ 1144us) server_negotiation.cc:407] Connection header received
0806 02:50:59.285714 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.285718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.285777 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.285860 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.287209 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.287776 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.289005 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.289250 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.289389 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.292317 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.292320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.292323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.292352 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.293807 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.294531 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.294535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.294537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.294689 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.295187 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.295190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.295192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.295519 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.295632 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.296360 (+ 728us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.296534 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:50:59.297560 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.283175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:35649)
0806 02:50:59.283799 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.283905 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.285480 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.286052 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.286060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.286461 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.287040 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.287054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.287924 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.287927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.288863 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.288873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.289443 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.289449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.289632 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.290346 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.290364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.292098 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.293951 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.293958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.293979 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.294364 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.294851 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.294855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.294856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.295057 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.295638 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.295644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.296035 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.296259 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.296638 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":182,"threads_started":1}
I20260806 02:50:59.298959 1998 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44675
I20260806 02:50:59.299226 1998 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:59.299782 1998 heartbeater.cc:511] Master 127.24.130.190:44675 requested a full tablet report, sending...
I20260806 02:50:59.300915 1352 ts_manager.cc:194] Registered new tserver with Master: 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:50:59.301514 1352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:35649
I20260806 02:50:59.308889 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.130.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.189.keytab.
Entry for principal oryx/127.24.130.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.189.keytab.
I20260806 02:50:59.342396 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:39919
--webserver_interface=127.24.130.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.189.keytab
--principal=oryx/127.24.130.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44675,127.24.130.189:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:59.466424 2008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:59.466802 2008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:59.466848 2008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:59.471063 2008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:59.471143 2008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:59.471163 2008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:59.471181 2008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:59.471197 2008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:59.477144 2008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44675,127.24.130.189:39919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.189.keytab
--principal=oryx/127.24.130.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.130.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
W20260806 02:50:59.477454 2008 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.130.190:44675,127.24.130.189:39919'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:50:59.478515 2008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:59.479676 2008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:59.489248 2014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.489301 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.489257 2013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:59.489506 2008 server_base.cc:1034] running on GCE node
I20260806 02:50:59.490279 2008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:59.491714 2008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:59.492931 2008 hybrid_clock.cc:648] HybridClock initialized: now 1785984659492912 us; error 38 us; skew 500 ppm
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:59.496136 2008 init.cc:377] Logged in from keytab as oryx/127.24.130.189@KRBTEST.COM (short username oryx)
I20260806 02:50:59.497268 2008 webserver.cc:460] Webserver started at http://127.24.130.189:37961/ using document root <none> and password file <none>
I20260806 02:50:59.497903 2008 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:59.498008 2008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:59.498272 2008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:59.500712 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/instance:
uuid: "cc865560c44b4066b5a4d064e91f878f"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "4eb424b7dcc71b3862d6f035a71d2981"
server_key_iv: "0826c9b5ea70d07433a27457290485fd"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.501387 2008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal/instance:
uuid: "cc865560c44b4066b5a4d064e91f878f"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "4eb424b7dcc71b3862d6f035a71d2981"
server_key_iv: "0826c9b5ea70d07433a27457290485fd"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.505761 2008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:59.508759 2029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.510154 2008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:59.510339 2008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
uuid: "cc865560c44b4066b5a4d064e91f878f"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "4eb424b7dcc71b3862d6f035a71d2981"
server_key_iv: "0826c9b5ea70d07433a27457290485fd"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.510489 2008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:59.527750 2008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:59.531522 2008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:59.531776 2008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:59.539624 2008 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.189:39919
I20260806 02:50:59.539952 2096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.189:39919 every 8 connection(s)
I20260806 02:50:59.540766 2008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
I20260806 02:50:59.541280 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 2008
I20260806 02:50:59.544224 2098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:59.549121 2098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.189@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:50:59.558789 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.543150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.24.130.189:39919)
0806 02:50:59.543606 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.543629 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.543669 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:50:59.544423 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.544456 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.544820 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.545143 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.546214 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.547385 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.548667 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.549162 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.549298 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.552763 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.552769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.552778 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.552806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.554950 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.555546 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.555555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.555563 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.555736 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.556106 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.556109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.556111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.556766 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.557052 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.557298 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.557644 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":150,"threads_started":1}
I20260806 02:50:59.562130 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.551563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.24.130.190:44675)
0806 02:50:59.551765 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.551770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.552106 (+ 336us) server_negotiation.cc:407] Connection header received
0806 02:50:59.552345 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.552347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.552395 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.552468 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.553291 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.553839 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.554857 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.555151 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.555278 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.558183 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.558187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.558190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.558225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.559989 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.560562 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.560564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.560566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.560652 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.560958 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.560964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.560969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.561251 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.561395 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.561797 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.561956 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:50:59.562141 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.551499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:42572)
0806 02:50:59.551961 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.552024 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.552215 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.552563 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.552572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.553021 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.553153 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.553159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.553962 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.553967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.554720 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.554731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.555298 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.555301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.555412 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.556031 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.556043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.558007 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.560121 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.560129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.560134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.560451 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.560745 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.560748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.560749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.560849 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.561414 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.561420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.561559 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.561727 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.561940 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":132,"threads_started":1}
I20260806 02:50:59.563398 2098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.189@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
I20260806 02:50:59.572187 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.563963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49824)
0806 02:50:59.564094 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.564111 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.564190 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.564427 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.564429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.564574 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.564671 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.564675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.565789 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.565792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.566311 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.566319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.566846 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.566849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.566930 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.567414 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.567427 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.568805 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.570718 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.570722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.570725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.570957 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.571221 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.571223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.571224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.571304 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.571667 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.571684 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.571786 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.571878 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.572008 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:50:59.572192 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.564121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.24.130.189:39919)
0806 02:50:59.564247 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.564250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.564262 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:50:59.564292 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.564294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.564329 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.564402 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.564769 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.565659 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.566429 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.566719 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.566802 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.569175 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.569178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.569180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.569201 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.570542 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.571053 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.571056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.571057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.571133 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.571382 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.571385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.571386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.571575 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.571653 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.571937 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.572068 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":52}
I20260806 02:50:59.575614 2098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Bootstrap starting.
I20260806 02:50:59.578318 2098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:59.579191 2098 log.cc:824] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:59.581410 2098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: No bootstrap required, opened a new log
I20260806 02:50:59.584905 2098 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } }
I20260806 02:50:59.585170 2098 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:59.585246 2098 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc865560c44b4066b5a4d064e91f878f, State: Initialized, Role: FOLLOWER
I20260806 02:50:59.585760 2098 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } }
I20260806 02:50:59.586371 2108 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } } }
I20260806 02:50:59.586536 2108 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:59.587172 2098 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:50:59.592430 2121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:59.592555 2121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:59.594623 2008 master_runner.cc:428] Detected that this master cc865560c44b4066b5a4d064e91f878f is joining an existing cluster
I20260806 02:50:59.594748 2008 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.130.189:39919
I20260806 02:50:59.606855 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.597987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.24.130.189:39919)
0806 02:50:59.598094 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.598098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.598158 (+ 60us) server_negotiation.cc:407] Connection header received
0806 02:50:59.598319 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.598322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.598359 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.598438 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.599109 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.599896 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.601075 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.601416 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.601490 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.603420 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.603426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.603431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.603460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.604940 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605488 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.605492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.605493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.605572 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605859 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.605862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.605864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.606097 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.606178 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.606493 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.606738 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":49}
I20260806 02:50:59.606869 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.597661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49834)
0806 02:50:59.598061 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.598076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.598208 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.598478 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.598481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.598795 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.598973 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.598982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.600043 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.600046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.600947 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.600958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.601635 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.601638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.601730 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.602810 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.602825 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.603268 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.605060 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.605066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.605397 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.605651 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.605658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.605747 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.606244 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.606247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.606330 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.606445 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.606619 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":119,"threads_started":1}
I20260806 02:50:59.607298 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.597643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.24.130.190:44675)
0806 02:50:59.597789 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.597794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.598086 (+ 292us) server_negotiation.cc:407] Connection header received
0806 02:50:59.598304 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.598308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.598364 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.598453 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.599123 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.599675 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.600582 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.600822 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.600954 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.602425 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.602431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.602436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.602475 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.604328 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605402 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.605406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.605408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.605509 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605844 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.605849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.605851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.606097 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.606204 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.606540 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.606966 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:50:59.607995 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.597571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:42582)
0806 02:50:59.597977 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.597993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.598180 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.598577 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.598579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.598862 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.598974 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.598980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.599796 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.599803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.600445 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.600454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.600969 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.600972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.601067 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.601705 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.601717 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.602261 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.605046 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.605049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.605282 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.605615 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.605618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.605620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.605722 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.606207 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.606210 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.606328 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.607635 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.607816 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.190@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
I20260806 02:50:59.624240 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.613278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49842)
0806 02:50:59.613697 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.613754 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.613932 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.614292 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.614297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.614879 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.615047 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.615060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.616230 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.616233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.617160 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.617171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.617665 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.617668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.617794 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.618508 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.618521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.620169 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.622061 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.622072 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.622080 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.622409 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.622750 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.622755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.622757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.622854 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.623400 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.623409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.623569 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.623751 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.624037 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260806 02:50:59.624255 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.613290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.24.130.189:39919)
0806 02:50:59.613440 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.613445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.613829 (+ 384us) server_negotiation.cc:407] Connection header received
0806 02:50:59.614096 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.614098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.614145 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.614253 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.615217 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.616064 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.617299 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.617527 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.617643 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.620336 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.620339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.620342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.620378 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.621928 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.622520 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.622524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.622526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.622614 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.622960 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.622966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.622971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.623256 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.623376 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.623843 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.624009 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:50:59.626628 1352 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.130.189:39919: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cc865560c44b4066b5a4d064e91f878f"
member_type: NON_VOTER
last_known_addr {
host: "127.24.130.189"
port: 39919
}
attrs {
promote: true
}
}
dest_uuid: "f26b4ce127e949b59618b96a68c890de"
cas_config_opid_index: -1
I20260806 02:50:59.627638 1352 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.24.130.188@KRBTEST.COM; defaulting to no policy
I20260806 02:50:59.630630 2062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.188@KRBTEST.COM" created.
I20260806 02:50:59.630342 2137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cc865560c44b4066b5a4d064e91f878f (127.24.130.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:50:59.632457 2062 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Refusing update from remote peer f26b4ce127e949b59618b96a68c890de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 02:50:59.633080 2137 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.130.189:39919
I20260806 02:50:59.633571 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:50:59.633668 2143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:59.633824 2008 master.cc:537] Master@127.24.130.189:39919 shutting down...
I20260806 02:50:59.633793 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:59.637077 1331 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 02:50:59.638332 1331 consensus_peers.cc:487] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de -> Peer cc865560c44b4066b5a4d064e91f878f (127.24.130.189:39919): Couldn't send request to peer cc865560c44b4066b5a4d064e91f878f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:50:59.639312 2008 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:59.639694 2008 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:59.639878 2008 tablet_replica.cc:330] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.188.keytab.
Entry for principal oryx/127.24.130.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.188.keytab.
I20260806 02:50:59.655325 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:38599
--webserver_interface=127.24.130.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.188.keytab
--principal=oryx/127.24.130.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:59.655987 2008 master.cc:552] Master@127.24.130.189:39919 shutdown complete.
I20260806 02:50:59.656054 2008 master_runner.cc:305] Clearing existing system tablet
I20260806 02:50:59.656335 2008 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:59.656399 2008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:59.657894 2008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 02:50:59.658967 2149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.659255 2008 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:59.659397 2008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:59.659432 2008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
uuid: "cc865560c44b4066b5a4d064e91f878f"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "4eb424b7dcc71b3862d6f035a71d2981"
server_key_iv: "0826c9b5ea70d07433a27457290485fd"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.661798 2008 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:59.664605 2008 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:50:59.664719 2008 log.cc:1196] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 02:50:59.665004 2008 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Deleting consensus metadata
I20260806 02:50:59.665190 2008 master_runner.cc:315] Copying system tablet from 127.24.130.190:44675
I20260806 02:50:59.666703 2008 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.130.190:44675
I20260806 02:50:59.680424 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.667561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.24.130.190:44675)
0806 02:50:59.667703 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.667708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.671084 (+ 3376us) server_negotiation.cc:407] Connection header received
0806 02:50:59.671359 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.671363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.671430 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.671540 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.672223 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.673099 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.674384 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.674795 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.674917 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.676451 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.676456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.676459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.676494 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.678381 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.678952 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.678957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.678962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.679067 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.679404 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.679410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.679414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.679675 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.679782 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.680104 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.680242 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:50:59.680431 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.667447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:42598)
0806 02:50:59.670925 (+ 3478us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.670947 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.671201 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.671587 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.671590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.671868 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.672032 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.672044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.673239 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.673243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.674195 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.674206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.674942 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.674946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.675054 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.675745 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.675762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.676266 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.678511 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.678515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.678517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.678828 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.679172 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.679177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.679181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.679289 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.679790 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.679793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.679910 (+ 117us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.680024 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.680189 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3371,"thread_start_us":110,"threads_started":1}
I20260806 02:50:59.681711 1377 tablet_copy_service.cc:140] P f26b4ce127e949b59618b96a68c890de: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cc865560c44b4066b5a4d064e91f878f ({username='oryx', principal='oryx/127.24.130.189@KRBTEST.COM'} at 127.0.0.1:42598)
I20260806 02:50:59.681953 1377 tablet_copy_service.cc:161] P f26b4ce127e949b59618b96a68c890de: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cc865560c44b4066b5a4d064e91f878f at {username='oryx', principal='oryx/127.24.130.189@KRBTEST.COM'} at 127.0.0.1:42598: session id = cc865560c44b4066b5a4d064e91f878f-00000000000000000000000000000000
I20260806 02:50:59.684196 1377 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:50:59.686586 2008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:50:59.689551 2008 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: tablet copy: Starting download of 0 data blocks...
I20260806 02:50:59.689795 2008 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: tablet copy: Starting download of 1 WAL segments...
I20260806 02:50:59.692174 2008 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:50:59.694748 1377 tablet_copy_service.cc:342] P f26b4ce127e949b59618b96a68c890de: Request end of tablet copy session cc865560c44b4066b5a4d064e91f878f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.130.189@KRBTEST.COM'} at 127.0.0.1:42598
I20260806 02:50:59.695052 1377 tablet_copy_service.cc:433] P f26b4ce127e949b59618b96a68c890de: ending tablet copy session cc865560c44b4066b5a4d064e91f878f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cc865560c44b4066b5a4d064e91f878f
I20260806 02:50:59.704753 2008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:59.706831 2158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.707175 2161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:59.707425 2008 server_base.cc:1034] running on GCE node
W20260806 02:50:59.707531 2159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:59.707751 2008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:59.708169 2008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:59.709384 2008 hybrid_clock.cc:648] HybridClock initialized: now 1785984659709363 us; error 58 us; skew 500 ppm
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:59.711436 2008 init.cc:377] Logged in from keytab as oryx/127.24.130.189@KRBTEST.COM (short username oryx)
I20260806 02:50:59.712067 2008 webserver.cc:460] Webserver started at http://127.24.130.189:46757/ using document root <none> and password file <none>
I20260806 02:50:59.712322 2008 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:59.712419 2008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:59.713958 2008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:59.715255 2173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.715569 2008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:50:59.715689 2008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
uuid: "cc865560c44b4066b5a4d064e91f878f"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "4eb424b7dcc71b3862d6f035a71d2981"
server_key_iv: "0826c9b5ea70d07433a27457290485fd"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.715855 2008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:59.736457 2008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:59.739693 2008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:59.745853 2008 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.189:39919
I20260806 02:50:59.746043 2253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.189:39919 every 8 connection(s)
I20260806 02:50:59.746801 2008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
I20260806 02:50:59.748893 2256 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:50:59.750191 2256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Bootstrap starting.
I20260806 02:50:59.761467 2256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:50:59.761880 2256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Bootstrap complete.
I20260806 02:50:59.762547 2256 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } }
I20260806 02:50:59.762732 2256 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc865560c44b4066b5a4d064e91f878f, State: Initialized, Role: LEARNER
I20260806 02:50:59.763015 2256 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } }
I20260806 02:50:59.763477 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:50:59.763602 2256 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:59.763702 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: LEARNER
I20260806 02:50:59.767424 2277 catalog_manager.cc:1259] Loaded cluster ID: 203476151937425f84de8a39b4accf79
I20260806 02:50:59.767561 2277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: loading cluster ID for follower catalog manager: success
I20260806 02:50:59.769932 2277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: acquiring CA information for follower catalog manager: success
I20260806 02:50:59.771798 2277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:50:59.802210 1617 heartbeater.cc:503] Master 127.24.130.190:44675 was elected leader, sending a full tablet report...
W20260806 02:50:59.812887 2147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:59.813265 2147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:59.813355 2147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:59.817976 2147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:59.818089 2147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:59.818125 2147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:59.818193 2147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:59.818239 2147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:59.824280 2147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.188.keytab
--principal=oryx/127.24.130.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:38599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.130.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:59.825650 2147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:59.826982 2147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:59.836265 2286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.836249 2285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:59.836256 2289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:59.836568 2147 server_base.cc:1034] running on GCE node
I20260806 02:50:59.837268 2147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:59.838538 2147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:59.839749 2147 hybrid_clock.cc:648] HybridClock initialized: now 1785984659839735 us; error 32 us; skew 500 ppm
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:59.842823 2147 init.cc:377] Logged in from keytab as oryx/127.24.130.188@KRBTEST.COM (short username oryx)
I20260806 02:50:59.843837 2147 webserver.cc:460] Webserver started at http://127.24.130.188:33089/ using document root <none> and password file <none>
I20260806 02:50:59.844377 2147 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:59.844439 2147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:59.844626 2147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:59.846856 2147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/instance:
uuid: "5101aa9224954066bca7be6cb8e9b1d0"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "d0142824edc7a6190bb66fb8559896b5"
server_key_iv: "01b807c73c138afc098bc61c6dedb2db"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.847411 2147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal/instance:
uuid: "5101aa9224954066bca7be6cb8e9b1d0"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "d0142824edc7a6190bb66fb8559896b5"
server_key_iv: "01b807c73c138afc098bc61c6dedb2db"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.851564 2147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:59.854242 2301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:59.855885 2147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:59.856052 2147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
uuid: "5101aa9224954066bca7be6cb8e9b1d0"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "d0142824edc7a6190bb66fb8559896b5"
server_key_iv: "01b807c73c138afc098bc61c6dedb2db"
server_key_version: "encryptionkey@0"
I20260806 02:50:59.856187 2147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:59.870584 2147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:59.874006 2147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:59.874245 2147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:59.882519 2147 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.188:38599
I20260806 02:50:59.882844 2374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.188:38599 every 8 connection(s)
I20260806 02:50:59.884011 2147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
I20260806 02:50:59.885228 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 2147
I20260806 02:50:59.887786 2375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:59.892099 2375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.130.188@KRBTEST.COM
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.188@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:50:59.902170 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.887006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.24.130.188:38599)
0806 02:50:59.887487 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.887503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.887545 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:50:59.888302 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.888320 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.888682 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.889073 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.889996 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.891162 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.892119 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.892705 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.892894 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.896405 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.896411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.896421 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.896452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.898379 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.899257 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.899264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.899272 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.899477 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.899857 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.899865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.899869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.900390 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.900561 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.900970 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.901310 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"spinlock_wait_cycles":7424,"thread_start_us":151,"threads_started":1}
I20260806 02:50:59.905053 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.894229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:42604)
0806 02:50:59.894918 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.894988 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.895148 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.895525 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.895532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.896025 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.896187 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.896194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.897044 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.897047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.897755 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.897765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.898266 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.898270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.898418 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.899332 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.899348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.901106 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.903144 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.903152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.903157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.903465 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.903787 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.903789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.903790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.903868 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.904347 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.904353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.904477 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.904666 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.904875 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"spinlock_wait_cycles":1792,"thread_start_us":315,"threads_started":1}
I20260806 02:50:59.905140 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.894622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.24.130.190:44675)
0806 02:50:59.894800 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.894804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.895094 (+ 290us) server_negotiation.cc:407] Connection header received
0806 02:50:59.895264 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.895270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.895343 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.895424 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.896339 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.896893 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.897911 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.898145 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.898262 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.901259 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.901262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.901264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.901294 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.903032 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.903582 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.903586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.903587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.903688 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.903969 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.903972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.903974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.904213 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.904320 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.904909 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.905019 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 02:50:59.906180 2375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.188@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
I20260806 02:50:59.914623 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.906823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49850)
0806 02:50:59.906948 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.906961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.907029 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.907407 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.907408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.907579 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.907683 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.907688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.908594 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.908597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.909141 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.909148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.909585 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.909588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.909663 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.910220 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.910232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.911557 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.913296 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.913299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.913301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.913506 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.913761 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.913763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.913764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.913838 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.914186 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.914188 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.914272 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.914354 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.914481 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 02:50:59.914754 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.906910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.24.130.189:39919)
0806 02:50:59.907192 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.907196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.907209 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:50:59.907250 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.907252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.907293 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.907370 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.907813 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.908497 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.909248 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.909487 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.909569 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.911709 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.911712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.911714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.911761 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.913191 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.913595 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.913598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.913599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.913681 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.913918 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.913920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.913921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.914104 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.914194 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.914416 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.914528 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":122,"threads_started":1}
I20260806 02:50:59.915625 2375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.188@KRBTEST.COM for oryx/127.24.130.188@KRBTEST.COM
I20260806 02:50:59.924348 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.916179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.0.0.1:42846)
0806 02:50:59.916332 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.916349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.916424 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.916641 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.916643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.916778 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.916861 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.916866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.918112 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.918115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.918637 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.918708 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.919185 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.919188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.919261 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.919747 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.919762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.921069 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.922994 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.922997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.922999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.923186 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.923439 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.923441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.923442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.923520 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.923890 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.923892 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.923974 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.924056 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.924181 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 02:50:59.924364 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.916315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.24.130.188:38599)
0806 02:50:59.916448 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.916451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.916464 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:50:59.916494 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.916496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.916530 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.916595 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.917007 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.917530 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.918827 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.919084 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.919166 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.921224 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.921226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.921228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.921251 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.922891 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.923271 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.923274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.923275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.923352 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.923597 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.923600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.923601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.923788 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.923898 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.924092 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.924240 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":64}
I20260806 02:50:59.927987 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Bootstrap starting.
I20260806 02:50:59.930500 2375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:59.931430 2375 log.cc:824] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:59.933691 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: No bootstrap required, opened a new log
I20260806 02:50:59.937162 2375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } }
I20260806 02:50:59.937462 2375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:59.937556 2375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5101aa9224954066bca7be6cb8e9b1d0, State: Initialized, Role: FOLLOWER
I20260806 02:50:59.938050 2375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } }
I20260806 02:50:59.938753 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } } }
I20260806 02:50:59.938906 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:59.939534 2375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:50:59.944885 2400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:59.945019 2400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:59.949134 2147 master_runner.cc:428] Detected that this master 5101aa9224954066bca7be6cb8e9b1d0 is joining an existing cluster
I20260806 02:50:59.949245 2147 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.130.188:38599
I20260806 02:50:59.956385 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 1435
I20260806 02:50:59.965342 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.952323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.0.0.1:42860)
0806 02:50:59.953005 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.953018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.953124 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.953415 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.953418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.953673 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.954066 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.954076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.956251 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.956254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.957136 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.957147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.958161 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.958166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.958269 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.959183 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.959197 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.959716 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.963146 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.963150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.963152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.963448 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.963841 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.963844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.963846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.964004 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.964639 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.964642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.964777 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.964894 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.965112 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":105,"spinlock_wait_cycles":2176,"thread_start_us":118,"threads_started":1}
I20260806 02:50:59.965976 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.952728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.24.130.188:38599)
0806 02:50:59.952838 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.952844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.953119 (+ 275us) server_negotiation.cc:407] Connection header received
0806 02:50:59.953243 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.953246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.953292 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.953373 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.955340 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.956121 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.957275 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.957626 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.957697 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.959923 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.959930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.959936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.959974 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.962988 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.963581 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.963588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.963593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.963717 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.964118 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.964124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.964129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.964474 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.964606 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.965058 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.965231 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:50:59.983441 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.952821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.24.130.190:44675)
0806 02:50:59.952950 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.952954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.952976 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:50:59.953033 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.953039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.953104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.953184 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.954256 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.955180 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.975181 (+ 20001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.975566 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.975699 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.978113 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.978118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.978120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.978153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.981132 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.981735 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.981739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.981741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.981867 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.982212 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.982218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.982221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.982512 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.982589 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.983076 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.983238 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:50:59.984145 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.952313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.24.130.189:39919)
0806 02:50:59.952482 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.952488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.959092 (+ 6604us) server_negotiation.cc:407] Connection header received
0806 02:50:59.960033 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.960037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.960114 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.960170 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.975447 (+ 15277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.976270 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.977410 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.977799 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.977896 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.979780 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.979784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.979786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.979812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.981769 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.982280 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.982283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.982285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.982388 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.982810 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.982814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.982816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.983059 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:59.983146 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:59.983821 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 02:50:59.983986 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:50:59.984207 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.952248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:42614)
0806 02:50:59.952795 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.952815 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.952931 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.953274 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.953278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.953673 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.954067 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.954076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.955348 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.955353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.974868 (+ 19515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.974903 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.975938 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.975942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.976064 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.977338 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.977360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.977924 (+ 564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.981300 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.981303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.981305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.981586 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.981997 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.981999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.982001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.982090 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.982702 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.982707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.982833 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.983748 (+ 915us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.983988 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":120,"spinlock_wait_cycles":4352,"thread_start_us":147,"threads_started":1}
I20260806 02:50:59.984364 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:42361
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:44675
--webserver_port=37867
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:59.985545 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.952463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49852)
0806 02:50:59.958945 (+ 6482us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.958963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.959897 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.960589 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.960592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.975089 (+ 14497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.975265 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.975274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.976498 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.976501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.977202 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.977223 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.978108 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.978112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.978223 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.979078 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.979090 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.979607 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.981901 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.981907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.981910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.982162 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.982509 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.982512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.982514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.982601 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.983212 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:59.983216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:59.983310 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:50:59.985108 (+ 1798us) client_negotiation.cc:240] Negotiation successful
0806 02:50:59.985317 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6390,"thread_start_us":109,"threads_started":1}
Aug 06 02:50:59 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984658, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.190@KRBTEST.COM for oryx/127.24.130.188@KRBTEST.COM
I20260806 02:51:00.001287 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.989205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.0.0.1:42876)
0806 02:50:59.989404 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:59.989420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:59.989505 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:59.989858 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:59.989861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:59.990106 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:59.990362 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.990378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.991527 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.991534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.992623 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:59.992642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.993325 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.993333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:59.993491 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.994204 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:59.994225 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:59.996277 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:59.998881 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.998888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.998892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.999229 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:59.999664 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:59.999670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:59.999674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:59.999778 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.000605 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.000608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.000747 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.000881 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.001081 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260806 02:51:00.002077 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:59.989304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.24.130.188:38599)
0806 02:50:59.989455 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:59.989463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:59.989480 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:50:59.989633 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:59.989638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:59.989711 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:59.989817 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:59.990536 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.991379 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.992799 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:59.993159 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:59.993280 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:59.996509 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:59.996517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:59.996522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:59.996565 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:59.998688 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.999406 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.999411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.999414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:59.999523 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:59.999892 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:59.999895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:59.999897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.000232 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.000527 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.001765 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.001915 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:51:00.004987 1352 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.130.188:38599: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0"
member_type: NON_VOTER
last_known_addr {
host: "127.24.130.188"
port: 38599
}
attrs {
promote: true
}
}
dest_uuid: "f26b4ce127e949b59618b96a68c890de"
cas_config_opid_index: 5
I20260806 02:51:00.005715 1352 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } }
I20260806 02:51:00.007081 2137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.007598 2340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:00.008664 2137 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.130.188:38599
I20260806 02:51:00.008855 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.008951 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.009249 2340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Refusing update from remote peer f26b4ce127e949b59618b96a68c890de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:51:00.009989 2143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Connected to new peer: Peer: permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:00.011020 2147 master.cc:537] Master@127.24.130.188:38599 shutting down...
I20260806 02:51:00.014863 2340 raft_consensus.cc:858] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } }
W20260806 02:51:00.014992 2340 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } }
I20260806 02:51:00.017019 2340 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cc865560c44b4066b5a4d064e91f878f (127.24.130.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:51:00.019601 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26b4ce127e949b59618b96a68c890de. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:51:00.019745 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 02:51:00.019843 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:51:00.019917 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 02:51:00.019992 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } }
I20260806 02:51:00.020054 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 02:51:00.022413 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.010118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.24.130.189:39919)
0806 02:51:00.010258 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.010263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.016987 (+ 6724us) server_negotiation.cc:407] Connection header received
0806 02:51:00.017257 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.017261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.017327 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.017405 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.017953 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.018810 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.020751 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.021457 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.021570 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.021725 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.022147 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.022250 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:51:00.022913 2414 mvcc.cc:205] Tried to move back new op lower bound from 7315393161237086208 to 7315393161114931200. Current Snapshot: MvccSnapshot[applied={T|T < 7315393161155416064}]
I20260806 02:51:00.023164 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.009767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49868)
0806 02:51:00.016835 (+ 7068us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.016858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.017117 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.017687 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.017690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.017711 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.017818 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.017825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.018978 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.018982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.020569 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.020586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.021843 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.021847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.021953 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.022019 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.022860 (+ 841us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.022969 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6941}
I20260806 02:51:00.023947 2201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 LEARNER]: Refusing update from remote peer f26b4ce127e949b59618b96a68c890de: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:51:00.024474 2143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 02:51:00.026927 2201 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.029516 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26b4ce127e949b59618b96a68c890de. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.029676 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: LEARNER
I20260806 02:51:00.029737 2143 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: attempting to promote NON_VOTER cc865560c44b4066b5a4d064e91f878f to VOTER
I20260806 02:51:00.029778 2259 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: NON_VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: true } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.029847 2259 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: LEARNER
I20260806 02:51:00.030213 2143 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } }
I20260806 02:51:00.032356 2201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 LEARNER]: Refusing update from remote peer f26b4ce127e949b59618b96a68c890de: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260806 02:51:00.032567 1328 consensus_peers.cc:487] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de -> Peer 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188:38599): Couldn't send request to peer 5101aa9224954066bca7be6cb8e9b1d0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:51:00.032886 2137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 02:51:00.034847 2143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cc865560c44b4066b5a4d064e91f878f (127.24.130.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.035343 2201 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cc865560c44b4066b5a4d064e91f878f (127.24.130.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.035601 1809 heartbeater.cc:503] Master 127.24.130.190:44675 was elected leader, sending a full tablet report...
I20260806 02:51:00.036463 2143 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.036564 2143 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.036703 2143 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.036788 2143 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.038138 2427 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.038327 2427 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:00.040526 2147 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260806 02:51:00.040850 2147 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260806 02:51:00.040942 2147 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: stopping tablet replica
I20260806 02:51:00.057994 2147 master.cc:552] Master@127.24.130.188:38599 shutdown complete.
I20260806 02:51:00.058084 2147 master_runner.cc:305] Clearing existing system tablet
I20260806 02:51:00.058351 2147 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:00.058436 2147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:00.060038 2147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:00.061110 2429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.061398 2147 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:00.061528 2147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:00.061564 2147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
uuid: "5101aa9224954066bca7be6cb8e9b1d0"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "d0142824edc7a6190bb66fb8559896b5"
server_key_iv: "01b807c73c138afc098bc61c6dedb2db"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.063616 2147 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:00.066311 2147 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:51:00.066429 2147 log.cc:1196] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:51:00.066797 2147 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Deleting consensus metadata
I20260806 02:51:00.067018 2147 master_runner.cc:315] Copying system tablet from 127.24.130.190:44675
I20260806 02:51:00.068311 2147 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: tablet copy: Beginning tablet copy session from remote peer at address 127.24.130.190:44675
I20260806 02:51:00.078850 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.069073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.24.130.190:44675)
0806 02:51:00.069219 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.069224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.069412 (+ 188us) server_negotiation.cc:407] Connection header received
0806 02:51:00.069543 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.069546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.069601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.069705 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.070272 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.071199 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.072395 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.072705 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.072818 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.074236 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.074241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.074246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.074280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.076579 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.077239 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.077243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.077245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.077357 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.077709 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.077712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.077714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.077981 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.078060 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.078461 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.078608 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:00.079747 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.068945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:42624)
0806 02:51:00.069288 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.069304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.069451 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.069747 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.069750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.069996 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.070124 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.070132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.071356 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.071360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.072247 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.072259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.072843 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.072847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.072937 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.073579 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.073594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.074067 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.076749 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.076752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.076754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.077053 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.077492 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.077497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.077502 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.077601 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.078141 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.078144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.078265 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.079367 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.079559 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260806 02:51:00.080554 1377 tablet_copy_service.cc:140] P f26b4ce127e949b59618b96a68c890de: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5101aa9224954066bca7be6cb8e9b1d0 ({username='oryx', principal='oryx/127.24.130.188@KRBTEST.COM'} at 127.0.0.1:42624)
I20260806 02:51:00.080672 1377 tablet_copy_service.cc:161] P f26b4ce127e949b59618b96a68c890de: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5101aa9224954066bca7be6cb8e9b1d0 at {username='oryx', principal='oryx/127.24.130.188@KRBTEST.COM'} at 127.0.0.1:42624: session id = 5101aa9224954066bca7be6cb8e9b1d0-00000000000000000000000000000000
I20260806 02:51:00.081892 1377 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:51:00.083475 2147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:51:00.086376 2147 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: tablet copy: Starting download of 0 data blocks...
I20260806 02:51:00.086624 2147 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: tablet copy: Starting download of 1 WAL segments...
I20260806 02:51:00.088783 2147 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:51:00.091509 1377 tablet_copy_service.cc:342] P f26b4ce127e949b59618b96a68c890de: Request end of tablet copy session 5101aa9224954066bca7be6cb8e9b1d0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.130.188@KRBTEST.COM'} at 127.0.0.1:42624
I20260806 02:51:00.091630 1377 tablet_copy_service.cc:433] P f26b4ce127e949b59618b96a68c890de: ending tablet copy session 5101aa9224954066bca7be6cb8e9b1d0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5101aa9224954066bca7be6cb8e9b1d0
I20260806 02:51:00.101811 2147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:00.104564 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:00.104592 2441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.105379 2147 server_base.cc:1034] running on GCE node
W20260806 02:51:00.105592 2443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.105911 2147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:00.106300 2147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:00.107474 2147 hybrid_clock.cc:648] HybridClock initialized: now 1785984660107452 us; error 49 us; skew 500 ppm
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:00.109395 2147 init.cc:377] Logged in from keytab as oryx/127.24.130.188@KRBTEST.COM (short username oryx)
I20260806 02:51:00.109988 2147 webserver.cc:460] Webserver started at http://127.24.130.188:44155/ using document root <none> and password file <none>
I20260806 02:51:00.110215 2147 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:00.110273 2147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:00.112001 2147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 02:51:00.113189 2456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.113628 2147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 02:51:00.113718 2147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
uuid: "5101aa9224954066bca7be6cb8e9b1d0"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "d0142824edc7a6190bb66fb8559896b5"
server_key_iv: "01b807c73c138afc098bc61c6dedb2db"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.113816 2147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:00.126463 2147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:00.129017 2147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:00.134481 2147 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.188:38599
I20260806 02:51:00.134506 2519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.188:38599 every 8 connection(s)
I20260806 02:51:00.135241 2147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
I20260806 02:51:00.137266 2520 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:51:00.138391 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Bootstrap starting.
I20260806 02:51:00.147622 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:51:00.147997 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: Bootstrap complete.
I20260806 02:51:00.148653 2520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } }
I20260806 02:51:00.148835 2520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5101aa9224954066bca7be6cb8e9b1d0, State: Initialized, Role: LEARNER
W20260806 02:51:00.148697 2413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260806 02:51:00.148952 2520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } }
W20260806 02:51:00.149102 2413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:00.149192 2413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:51:00.149518 2520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:00.152482 2550 catalog_manager.cc:1259] Loaded cluster ID: 203476151937425f84de8a39b4accf79
I20260806 02:51:00.152622 2550 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: loading cluster ID for follower catalog manager: success
I20260806 02:51:00.153482 2529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.153610 2529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:51:00.155288 2413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:00.155373 2413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:00.155462 2413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:00.155592 2550 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: acquiring CA information for follower catalog manager: success
I20260806 02:51:00.157101 2550 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:51:00.161827 2413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:42361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=37867
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:00.163550 2413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:00.164918 2413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:00.175596 2552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:00.175596 2553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.175843 2413 server_base.cc:1034] running on GCE node
W20260806 02:51:00.175596 2556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.176484 2413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:00.177246 2413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:00.178454 2413 hybrid_clock.cc:648] HybridClock initialized: now 1785984660178435 us; error 39 us; skew 500 ppm
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:00.181617 2413 init.cc:377] Logged in from keytab as oryx/127.24.130.129@KRBTEST.COM (short username oryx)
I20260806 02:51:00.182778 2413 webserver.cc:460] Webserver started at http://127.24.130.129:37867/ using document root <none> and password file <none>
I20260806 02:51:00.183411 2413 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:00.183544 2413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:00.188014 2413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:00.190477 2565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.191947 2413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:00.192111 2413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "3503ae3c442040b4af4744a432245e77"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "43fdf3e5f98a1a44b4e329318116a09b"
server_key_iv: "2aa47e62383f6cd83401a7ae73f2178b"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.192747 2413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:00.209157 2413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:00.212726 2413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:00.212968 2413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:00.213641 2413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:00.214730 2413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:00.214802 2413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.214885 2413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:00.214946 2413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.224819 2413 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:42361
I20260806 02:51:00.224850 2723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:42361 every 8 connection(s)
I20260806 02:51:00.225999 2413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:00.235743 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 2413
I20260806 02:51:00.236018 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 1626
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.188@KRBTEST.COM
I20260806 02:51:00.244222 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.228629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:35703 (local address 127.24.130.188:38599)
0806 02:51:00.228931 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.228936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.229357 (+ 421us) server_negotiation.cc:407] Connection header received
0806 02:51:00.230595 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.230601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.230691 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.230789 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.232272 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.233328 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.234833 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.235060 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.235193 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.240179 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.240182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.240184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.240214 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.241865 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.242463 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.242466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.242468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.242551 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.242964 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.242967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.242968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.243178 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.243239 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.243910 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.244051 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260806 02:51:00.245394 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.228031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:39423 (local address 127.24.130.189:39919)
0806 02:51:00.228185 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.228190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.229749 (+ 1559us) server_negotiation.cc:407] Connection header received
0806 02:51:00.230452 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.230457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.230530 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.230642 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.232252 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.232921 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.234771 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.235042 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.235173 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.239546 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.239553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.239560 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.239601 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.242682 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.243404 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.243411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.243416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.243543 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.243984 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.243991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.243996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.244318 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.244446 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.244945 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.245116 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:51:00.248965 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.228500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.24.130.129:35703)
0806 02:51:00.229184 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.229240 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.230223 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.230906 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.230913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.231529 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.232098 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.232114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.233443 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.233448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.234612 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.234624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.235251 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.235256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.235582 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.238651 (+ 3069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.238690 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.240006 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.242007 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.242013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.242032 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.242357 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.242686 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.242689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.242691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.242867 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.243311 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.243317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.243643 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.244621 (+ 978us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.247780 (+ 3159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":2985,"spinlock_wait_cycles":20224,"thread_start_us":731,"threads_started":1}
I20260806 02:51:00.250414 2724 heartbeater.cc:347] Connected to a master server at 127.24.130.188:38599
I20260806 02:51:00.250711 2724 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.251233 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.228483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.24.130.129:39423)
0806 02:51:00.229641 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.229656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.230240 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.230908 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.230917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.231525 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.232094 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.232109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.233070 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.233074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.234630 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.234642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.235312 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.235314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.235591 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.236597 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.236627 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.239266 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.242859 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.242866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.242871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.243226 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.243695 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.243701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.243706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.243833 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.244526 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.244532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.244675 (+ 143us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.245592 (+ 917us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.250929 (+ 5337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":5359,"spinlock_wait_cycles":35712,"thread_start_us":181,"threads_started":1}
I20260806 02:51:00.251331 2724 heartbeater.cc:511] Master 127.24.130.188:38599 requested a full tablet report, sending...
I20260806 02:51:00.252089 2728 heartbeater.cc:347] Connected to a master server at 127.24.130.189:39919
I20260806 02:51:00.252197 2728 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.252421 2728 heartbeater.cc:511] Master 127.24.130.189:39919 requested a full tablet report, sending...
I20260806 02:51:00.253078 2467 ts_manager.cc:194] Registered new tserver with Master: 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361)
I20260806 02:51:00.253590 2195 ts_manager.cc:194] Registered new tserver with Master: 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361)
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:00.258041 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:35157
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:44675
--webserver_port=38627
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:00.260438 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.228100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:56605 (local address 127.24.130.190:44675)
0806 02:51:00.228241 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.228246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.229780 (+ 1534us) server_negotiation.cc:407] Connection header received
0806 02:51:00.230952 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.230955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.231011 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.231106 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.232272 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.233098 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.235307 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.235674 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.235792 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.256111 (+ 20319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.256119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.256126 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.256170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.258393 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.258994 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.259001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.259006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.259113 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.259432 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.259437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.259442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.259733 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.259844 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.260153 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.260285 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:51:00.260803 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.228483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:56605)
0806 02:51:00.229690 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.229705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.230240 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.231142 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.231145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.231533 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.232098 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.232114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.233210 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.233213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.234630 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.234642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.236037 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.236040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.247335 (+ 11295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.251752 (+ 4417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.251773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.255903 (+ 4130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.258550 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.258553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.258555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.258876 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.259225 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.259227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.259229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.259324 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.259840 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.259842 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.259939 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.260054 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.260233 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":462,"spinlock_wait_cycles":78208,"thread_start_us":120,"threads_started":1}
I20260806 02:51:00.261595 2725 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44675
I20260806 02:51:00.261714 2725 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.261948 2725 heartbeater.cc:511] Master 127.24.130.190:44675 requested a full tablet report, sending...
I20260806 02:51:00.262974 1352 ts_manager.cc:194] Re-registered known tserver with Master: 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361)
I20260806 02:51:00.263942 1352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:56605
I20260806 02:51:00.303860 1998 heartbeater.cc:503] Master 127.24.130.190:44675 was elected leader, sending a full tablet report...
W20260806 02:51:00.383527 2735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:00.383893 2735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:00.383975 2735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:00.388253 2735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:00.388389 2735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:00.388494 2735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:00.394274 2735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:35157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=38627
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:00.395623 2735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:00.396792 2735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:00.407856 2741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.407975 2735 server_base.cc:1034] running on GCE node
W20260806 02:51:00.407859 2746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:00.408248 2743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.408730 2735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:00.409456 2735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:00.410689 2735 hybrid_clock.cc:648] HybridClock initialized: now 1785984660410630 us; error 41 us; skew 500 ppm
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:00.413956 2735 init.cc:377] Logged in from keytab as oryx/127.24.130.130@KRBTEST.COM (short username oryx)
I20260806 02:51:00.415022 2735 webserver.cc:460] Webserver started at http://127.24.130.130:38627/ using document root <none> and password file <none>
I20260806 02:51:00.415652 2735 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:00.415814 2735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:00.420055 2735 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:00.422302 2754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.423656 2735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:51:00.423804 2735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "30a374a29e454d9f9ef138d36e35e151"
format_stamp: "Formatted at 2026-08-06 02:50:58 on dist-test-slave-3kk6"
server_key: "30c762a443b71c27b0f89407d5845678"
server_key_iv: "bfcf784fb737acea69844c9655bf5015"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.424417 2735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:00.445545 2735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:00.449599 2735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:00.449813 2735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:00.450568 2735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:00.451753 2735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:00.451812 2735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.451871 2735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:00.451902 2735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.462942 2735 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:35157
I20260806 02:51:00.462987 2898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:35157 every 8 connection(s)
I20260806 02:51:00.464303 2735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:00.468495 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 2735
I20260806 02:51:00.468724 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 1817
I20260806 02:51:00.483009 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:33791
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:44675
--webserver_port=44793
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.188@KRBTEST.COM
I20260806 02:51:00.491444 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.467537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:51257 (local address 127.24.130.188:38599)
0806 02:51:00.467685 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.467690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.474031 (+ 6341us) server_negotiation.cc:407] Connection header received
0806 02:51:00.475066 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.475073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.475145 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.475205 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.478059 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.479046 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481472 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.481938 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.482041 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.486989 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.486995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.487002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.487048 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.489176 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.489711 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.489715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.489717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.489814 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.490151 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.490157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.490161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.490532 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.490613 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.491038 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.491204 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:51:00.491710 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.466547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:41289 (local address 127.24.130.190:44675)
0806 02:51:00.466732 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.466738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.468629 (+ 1891us) server_negotiation.cc:407] Connection header received
0806 02:51:00.475179 (+ 6550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.475185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.475240 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.475290 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.477914 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.478809 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481220 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.481589 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481673 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.486551 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.486556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.486559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.486594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.489575 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.490097 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.490103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.490107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.490206 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.490684 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.490688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.490690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.490952 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.491047 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.491402 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.491534 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:51:00.492705 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.467357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.24.130.130:33219)
0806 02:51:00.474927 (+ 7570us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.474947 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.475148 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.475734 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.475742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.476589 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.477679 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.477705 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.479036 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.479039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.481063 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.481070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.481915 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.482305 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.483524 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.483541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.485666 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.488250 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.488261 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.488286 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.488731 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.489129 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.489135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.489141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.489385 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.490043 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.490052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.490605 (+ 553us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.490925 (+ 320us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.491387 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7474,"mutex_wait_us":917,"spinlock_wait_cycles":36480,"thread_start_us":114,"threads_started":1}
I20260806 02:51:00.492705 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.467149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:41289)
0806 02:51:00.468444 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.468489 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.475043 (+ 6554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.475730 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.475741 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.476594 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.477679 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.477705 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.479036 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.479039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.481040 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.481056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.481915 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.482306 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.483447 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.483472 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.486062 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.489704 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.489706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.489708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.489973 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.490437 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.490440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.490442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.490537 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.491123 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.491125 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.491216 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.492016 (+ 800us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.492154 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":653,"spinlock_wait_cycles":12672,"thread_start_us":793,"threads_started":1}
I20260806 02:51:00.492705 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.467149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.24.130.130:51257)
0806 02:51:00.473887 (+ 6738us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.473905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.474762 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.475730 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.475741 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.476589 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.477691 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.477706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.479225 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.479231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.481063 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.481078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.482253 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.482260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.482403 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.483355 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.483382 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.486785 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.489331 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.489335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.489337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.489592 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.489942 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.489945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.489947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.490045 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.490715 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.490718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.490816 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.491624 (+ 808us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.491792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6622,"mutex_wait_us":510,"spinlock_wait_cycles":22144,"thread_start_us":173,"threads_started":1}
I20260806 02:51:00.494318 2903 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44675
I20260806 02:51:00.494354 2900 heartbeater.cc:347] Connected to a master server at 127.24.130.188:38599
I20260806 02:51:00.494547 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.466547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:33219 (local address 127.24.130.189:39919)
0806 02:51:00.466798 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.466801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.475057 (+ 8256us) server_negotiation.cc:407] Connection header received
0806 02:51:00.475293 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.475300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.475369 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.475430 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.477910 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.478809 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481219 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.481589 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.481673 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.485924 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.485930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.485934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.485973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.488122 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.488867 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.488874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.488878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.488984 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.489522 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.489526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.489527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.489859 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.489974 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.494233 (+ 4259us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.494395 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260806 02:51:00.494563 2900 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.494565 2903 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.495175 2904 heartbeater.cc:347] Connected to a master server at 127.24.130.189:39919
I20260806 02:51:00.495280 2904 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.495272 2903 heartbeater.cc:511] Master 127.24.130.190:44675 requested a full tablet report, sending...
I20260806 02:51:00.495272 2900 heartbeater.cc:511] Master 127.24.130.188:38599 requested a full tablet report, sending...
I20260806 02:51:00.495512 2904 heartbeater.cc:511] Master 127.24.130.189:39919 requested a full tablet report, sending...
I20260806 02:51:00.496557 1352 ts_manager.cc:194] Re-registered known tserver with Master: 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157)
I20260806 02:51:00.496660 2195 ts_manager.cc:194] Registered new tserver with Master: 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157)
I20260806 02:51:00.496735 2467 ts_manager.cc:194] Registered new tserver with Master: 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157)
I20260806 02:51:00.497403 1352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:41289
W20260806 02:51:00.611720 2916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:00.612077 2916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:00.612164 2916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:00.616483 2916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:00.616585 2916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:00.616685 2916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:00.622366 2916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:33791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=44793
--tserver_master_addrs=127.24.130.190:44675,127.24.130.189:39919,127.24.130.188:38599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:00.623826 2916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:00.625298 2916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:51:00.627775 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.623701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.0.0.1:42880)
0806 02:51:00.624010 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.624027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.624159 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.624493 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.624495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.624510 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.624616 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.624621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.625637 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.625639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.626412 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.626421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.627364 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.627367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.627442 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.627466 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.627520 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.627625 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
I20260806 02:51:00.627784 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.623810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.24.130.188:38599)
0806 02:51:00.623980 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.623985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.624133 (+ 148us) server_negotiation.cc:407] Connection header received
0806 02:51:00.624275 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.624279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.624347 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.624473 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.624731 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.625514 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.626549 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.627228 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.627366 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.627513 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.627580 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.627656 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:51:00.628631 2478 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 02:51:00.628971 2529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26b4ce127e949b59618b96a68c890de. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.629127 2529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:51:00.636651 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:00.636654 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.636883 2916 server_base.cc:1034] running on GCE node
W20260806 02:51:00.636667 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.637651 2916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:00.638513 2916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:00.639757 2916 hybrid_clock.cc:648] HybridClock initialized: now 1785984660639754 us; error 50 us; skew 500 ppm
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:00.643143 2916 init.cc:377] Logged in from keytab as oryx/127.24.130.131@KRBTEST.COM (short username oryx)
I20260806 02:51:00.644439 2916 webserver.cc:460] Webserver started at http://127.24.130.131:44793/ using document root <none> and password file <none>
I20260806 02:51:00.645079 2916 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:00.645206 2916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:00.649513 2916 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:00.651999 2938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.653475 2916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:00.653620 2916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "17b9266ccbe346fc9d20626d3129cbfa"
format_stamp: "Formatted at 2026-08-06 02:50:59 on dist-test-slave-3kk6"
server_key: "f6c952ece7e9e9bd8abe367d9bdd24e2"
server_key_iv: "8b16ea19ac60a852514653bb6649161a"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.654273 2916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:00.673792 2916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:00.677899 2916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:00.678135 2916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:00.678802 2916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:00.679764 2916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:00.679816 2916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.679868 2916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:00.679884 2916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.689756 2916 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:33791
I20260806 02:51:00.689790 3106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:33791 every 8 connection(s)
I20260806 02:51:00.691022 2916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:00.693579 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 2916
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.188@KRBTEST.COM
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.189@KRBTEST.COM
Aug 06 02:51:00 dist-test-slave-3kk6 krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:00.710064 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.693719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:56611 (local address 127.24.130.188:38599)
0806 02:51:00.693862 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.693868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.694398 (+ 530us) server_negotiation.cc:407] Connection header received
0806 02:51:00.695888 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.695895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.695971 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.696065 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.698225 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.699040 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.701034 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.701439 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.701551 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.705215 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.705221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.705223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.705253 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.707314 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708049 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.708053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.708056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.708150 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708596 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.708599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.708601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.708868 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.708948 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.709720 (+ 772us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.709868 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":63}
I20260806 02:51:00.710772 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.692956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:48395 (local address 127.24.130.189:39919)
0806 02:51:00.693101 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.693105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.694912 (+ 1807us) server_negotiation.cc:407] Connection header received
0806 02:51:00.696129 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.696132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.696189 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.696266 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.698181 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.699069 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.702246 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.702599 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.702739 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.706181 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.706185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.706190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.706225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.708075 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708598 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.708601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.708603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.708685 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708997 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.709002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.709005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.709222 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.709303 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.710472 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.710597 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:51:00.711108 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.696561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.24.130.188:38599)
0806 02:51:00.699276 (+ 2715us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.699281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.699295 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:51:00.699340 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.699344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.699404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.699496 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.700489 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.701705 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.702870 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.703208 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.703300 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.705449 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.705453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.705456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.705485 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.708409 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.709204 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.709208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.709210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.709303 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.709655 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.709658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.709660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.709901 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.710030 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.710793 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.710934 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2619,"thread_start_us":94,"threads_started":1}
I20260806 02:51:00.711536 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.692961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:39085 (local address 127.24.130.190:44675)
0806 02:51:00.693101 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.693105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.695282 (+ 2177us) server_negotiation.cc:407] Connection header received
0806 02:51:00.696474 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.696477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.696531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.696612 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.698373 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.699168 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.701783 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.702985 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.703055 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.707633 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.707636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.707639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.707669 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.709418 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.709933 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.709940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.709945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.710036 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.710486 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.710489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.710491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.710785 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.710861 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.711251 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.711377 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":817,"server-negotiator.queue_time_us":57}
I20260806 02:51:00.711607 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.693486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.24.130.131:56611)
0806 02:51:00.694214 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.694284 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.695591 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.696415 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.696426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.697100 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.698042 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.698063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.699244 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.699250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.700895 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.700908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.701625 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.701631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.701908 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.702700 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.702729 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.704985 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.707467 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.707475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.707500 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.707912 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.708263 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.708268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.708475 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.709033 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.709042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.709413 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.710247 (+ 834us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.710404 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":173,"spinlock_wait_cycles":4352,"thread_start_us":720,"threads_started":1}
I20260806 02:51:00.711995 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.693486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.24.130.131:48395)
0806 02:51:00.694783 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.694798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.695974 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.696491 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.696494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.697893 (+ 1399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.698041 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.698065 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.699276 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.699280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.702094 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.702104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.703180 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.703185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.703291 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.703975 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.703997 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.705620 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.708200 (+ 2580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.708210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.708475 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.708793 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.708795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.708797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.708890 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.709372 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.709375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.709448 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.709643 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.710016 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":1364,"spinlock_wait_cycles":2304,"thread_start_us":160,"threads_started":1}
I20260806 02:51:00.712195 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.693486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:39085)
0806 02:51:00.695155 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.695168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.696333 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.697037 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.697041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.697342 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:00.698254 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.698260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.700777 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.700780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.701603 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.701612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.703622 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.703626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.703736 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.705848 (+ 2112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:00.705861 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:00.707451 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:00.709540 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.709544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.709545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.709791 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.710160 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:00.710265 (+ 105us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:00.710270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:00.710369 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:00.710952 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:00.710955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:00.711045 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.711764 (+ 719us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.711939 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1582,"mutex_wait_us":1541,"thread_start_us":134,"threads_started":1}
I20260806 02:51:00.713274 3110 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44675
I20260806 02:51:00.713553 3110 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.713960 3111 heartbeater.cc:347] Connected to a master server at 127.24.130.189:39919
I20260806 02:51:00.714056 3111 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.714169 3110 heartbeater.cc:511] Master 127.24.130.190:44675 requested a full tablet report, sending...
I20260806 02:51:00.714236 3111 heartbeater.cc:511] Master 127.24.130.189:39919 requested a full tablet report, sending...
I20260806 02:51:00.714704 3109 heartbeater.cc:347] Connected to a master server at 127.24.130.188:38599
I20260806 02:51:00.714783 3109 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:00.714969 3109 heartbeater.cc:511] Master 127.24.130.188:38599 requested a full tablet report, sending...
I20260806 02:51:00.715595 2195 ts_manager.cc:194] Registered new tserver with Master: 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:00.716105 1352 ts_manager.cc:194] Re-registered known tserver with Master: 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:00.716614 2467 ts_manager.cc:194] Registered new tserver with Master: 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:00.716889 1352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:39085
I20260806 02:51:00.717270 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.696183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.24.130.190:44675)
0806 02:51:00.697964 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.697968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.697983 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:00.698022 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.698025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.698070 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.699791 (+ 1721us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.699897 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.700708 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.701836 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.702595 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.705268 (+ 2673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.705998 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.706003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.706007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.706040 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.714580 (+ 8540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.715143 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.715146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.715148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.715243 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.715726 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.715729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.715731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.715988 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.716123 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.716903 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.717055 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1585,"thread_start_us":77,"threads_started":1}
I20260806 02:51:00.721762 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.696238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.24.130.189:39919)
0806 02:51:00.698054 (+ 1816us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.698058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.704031 (+ 5973us) server_negotiation.cc:407] Connection header received
0806 02:51:00.705815 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.705820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.705883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.705954 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.713857 (+ 7903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.714765 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.715969 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.716264 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.716343 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.718211 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.718215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.718218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.718243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.720138 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.720593 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.720595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.720596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.720669 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.720951 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.720954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.720955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.721123 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.721216 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.721494 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.721596 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1734,"thread_start_us":108,"threads_started":1}
I20260806 02:51:00.720502 1352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:00.724454 1352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:00.727188 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: NON_VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: true } } }
I20260806 02:51:00.727346 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.729712 2137 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: attempting to promote NON_VOTER 5101aa9224954066bca7be6cb8e9b1d0 to VOTER
I20260806 02:51:00.730082 2137 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:00.731309 2201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Refusing update from remote peer f26b4ce127e949b59618b96a68c890de: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:51:00.731588 2478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 LEARNER]: Refusing update from remote peer f26b4ce127e949b59618b96a68c890de: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:51:00.731772 3138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:51:00.732260 3138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [LEADER]: Connected to new peer: Peer: permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:51:00.733759 3140 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.734282 2201 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.734495 2478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.736457 2529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.736605 2529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:00.737952 3149 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.738113 3149 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:00.738343 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.738482 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.738778 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.738893 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.739323 3140 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f26b4ce127e949b59618b96a68c890de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:00.739455 3140 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.748502 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.742430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:35157 (local address 127.0.0.1:35536)
0806 02:51:00.742545 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.742561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.742640 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.743694 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.743698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.743721 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.743843 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.743851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.745609 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.745616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.746755 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.746768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.748044 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.748047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.748174 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.748198 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.748268 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.748350 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":2}
I20260806 02:51:00.749086 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.742836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.24.130.130:35157)
0806 02:51:00.743147 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.743158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.743181 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:51:00.743250 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.743255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.743509 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.744030 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.744195 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.745486 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.746923 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.747712 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.747837 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.747981 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.748819 (+ 838us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.748941 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
I20260806 02:51:00.750593 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.743630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:33791 (local address 127.0.0.1:51340)
0806 02:51:00.743860 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.743875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.743969 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.745768 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.745772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.745801 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.745917 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.745932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.747063 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.747067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.748720 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.748731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.750038 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.750041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.750140 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.750161 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.750330 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.750448 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":520,"thread_start_us":101,"threads_started":1}
I20260806 02:51:00.750607 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.744040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.24.130.131:33791)
0806 02:51:00.745344 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.745360 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.745382 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:51:00.745450 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.745454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.745626 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.745825 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.746031 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.746915 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.748864 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.749727 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.749843 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.749967 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.750358 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.750461 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":105,"threads_started":1}
I20260806 02:51:00.752707 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.743276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42361 (local address 127.0.0.1:36302)
0806 02:51:00.743549 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.743566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.743653 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.747510 (+ 3857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.747514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.747533 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.747643 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.747649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.750062 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.750066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.751215 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.751226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.752312 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.752315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.752412 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.752448 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.752499 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.752575 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":123,"threads_started":1}
I20260806 02:51:00.752575 2814 tablet_service.cc:1467] Processing CreateTablet for tablet dc04216457194a4cb93feb5d9acc54ef (DEFAULT_TABLE table=test-table [id=a0bb2c5787014b2ab168fc82bd58b19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:00.753487 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.743472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.24.130.129:42361)
0806 02:51:00.746998 (+ 3526us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.747019 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.747047 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:51:00.747126 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.747131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.747341 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.748247 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.748415 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.749909 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.751390 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.752169 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.753076 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.753240 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.753289 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.753366 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3387,"thread_start_us":94,"threads_started":1}
I20260806 02:51:00.753790 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc04216457194a4cb93feb5d9acc54ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:00.753707 3011 tablet_service.cc:1467] Processing CreateTablet for tablet dc04216457194a4cb93feb5d9acc54ef (DEFAULT_TABLE table=test-table [id=a0bb2c5787014b2ab168fc82bd58b19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:00.754846 3011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc04216457194a4cb93feb5d9acc54ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:00.756058 2635 tablet_service.cc:1467] Processing CreateTablet for tablet dc04216457194a4cb93feb5d9acc54ef (DEFAULT_TABLE table=test-table [id=a0bb2c5787014b2ab168fc82bd58b19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:00.756954 2635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc04216457194a4cb93feb5d9acc54ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:00.765362 3165 tablet_bootstrap.cc:492] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Bootstrap starting.
I20260806 02:51:00.767081 3167 tablet_bootstrap.cc:492] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Bootstrap starting.
I20260806 02:51:00.767635 3165 tablet_bootstrap.cc:654] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:00.768457 3165 log.cc:824] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:00.769398 3167 tablet_bootstrap.cc:654] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:00.770277 3167 log.cc:824] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:00.770630 3165 tablet_bootstrap.cc:492] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: No bootstrap required, opened a new log
I20260806 02:51:00.770870 3165 ts_tablet_manager.cc:1397] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260806 02:51:00.772548 3167 tablet_bootstrap.cc:492] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: No bootstrap required, opened a new log
I20260806 02:51:00.772620 3172 tablet_bootstrap.cc:492] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Bootstrap starting.
I20260806 02:51:00.772753 3167 ts_tablet_manager.cc:1397] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:51:00.774876 3172 tablet_bootstrap.cc:654] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:00.775744 3172 log.cc:824] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:00.776222 3165 raft_consensus.cc:348] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.776557 3165 raft_consensus.cc:374] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:00.776623 3165 raft_consensus.cc:729] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9266ccbe346fc9d20626d3129cbfa, State: Initialized, Role: FOLLOWER
I20260806 02:51:00.777019 3167 raft_consensus.cc:348] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.777295 3167 raft_consensus.cc:374] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:00.777220 3165 consensus_queue.cc:260] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.777349 3167 raft_consensus.cc:729] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3503ae3c442040b4af4744a432245e77, State: Initialized, Role: FOLLOWER
I20260806 02:51:00.777788 3167 consensus_queue.cc:260] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.778349 3172 tablet_bootstrap.cc:492] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: No bootstrap required, opened a new log
I20260806 02:51:00.778410 3165 ts_tablet_manager.cc:1428] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 02:51:00.778543 3172 ts_tablet_manager.cc:1397] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 02:51:00.778613 3110 heartbeater.cc:503] Master 127.24.130.190:44675 was elected leader, sending a full tablet report...
I20260806 02:51:00.778769 3167 ts_tablet_manager.cc:1428] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:51:00.779126 2725 heartbeater.cc:503] Master 127.24.130.190:44675 was elected leader, sending a full tablet report...
I20260806 02:51:00.786973 3172 raft_consensus.cc:348] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.787281 3172 raft_consensus.cc:374] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:00.787376 3172 raft_consensus.cc:729] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a374a29e454d9f9ef138d36e35e151, State: Initialized, Role: FOLLOWER
I20260806 02:51:00.787892 3172 consensus_queue.cc:260] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.788990 3172 ts_tablet_manager.cc:1428] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:51:00.789541 2903 heartbeater.cc:503] Master 127.24.130.190:44675 was elected leader, sending a full tablet report...
I20260806 02:51:00.870718 3180 raft_consensus.cc:482] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:00.871022 3180 raft_consensus.cc:504] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.872309 3180 leader_election.cc:288] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361), 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:00.876437 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.872798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:41623 (local address 127.24.130.129:42361)
0806 02:51:00.872940 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.872943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.872961 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:51:00.873158 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.873162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.873215 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.873322 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.873672 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.874204 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.875371 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.875802 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.875963 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.876027 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.876226 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.876303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:51:00.876461 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.872674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42361 (local address 127.24.130.130:41623)
0806 02:51:00.872837 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.872856 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.873017 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.873369 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.873372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.873391 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.873534 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.873540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.874357 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.874361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.875211 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.875231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.875936 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.875939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.876068 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.876094 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.876196 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.876292 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:51:00.876657 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.873237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:38919 (local address 127.24.130.131:33791)
0806 02:51:00.873418 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.873430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.873442 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:51:00.873498 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.873500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.873540 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.873638 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:00.873951 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.874469 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.875602 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.876084 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.876228 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.876282 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.876460 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.876551 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:51:00.876654 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.873066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:33791 (local address 127.24.130.130:38919)
0806 02:51:00.873208 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:00.873224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:00.873323 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:00.873650 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:00.873653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:00.873669 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:00.873824 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.873829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.874600 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.874604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.875468 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:00.875486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.876224 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.876228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:00.876339 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.876363 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:51:00.876444 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:51:00.876523 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:51:00.877297 2664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc04216457194a4cb93feb5d9acc54ef" candidate_uuid: "30a374a29e454d9f9ef138d36e35e151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3503ae3c442040b4af4744a432245e77" is_pre_election: true
I20260806 02:51:00.877485 3042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc04216457194a4cb93feb5d9acc54ef" candidate_uuid: "30a374a29e454d9f9ef138d36e35e151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9266ccbe346fc9d20626d3129cbfa" is_pre_election: true
I20260806 02:51:00.877689 2664 raft_consensus.cc:2454] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a374a29e454d9f9ef138d36e35e151 in term 0.
I20260806 02:51:00.877847 3042 raft_consensus.cc:2454] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a374a29e454d9f9ef138d36e35e151 in term 0.
I20260806 02:51:00.878268 2758 leader_election.cc:302] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a374a29e454d9f9ef138d36e35e151, 3503ae3c442040b4af4744a432245e77; no voters:
I20260806 02:51:00.878571 3180 raft_consensus.cc:2781] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:00.878701 3180 raft_consensus.cc:482] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:00.878775 3180 raft_consensus.cc:3037] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:00.879976 3180 raft_consensus.cc:504] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.880364 3180 leader_election.cc:288] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [CANDIDATE]: Term 1 election: Requested vote from peers 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361), 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:00.880825 2664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc04216457194a4cb93feb5d9acc54ef" candidate_uuid: "30a374a29e454d9f9ef138d36e35e151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3503ae3c442040b4af4744a432245e77"
I20260806 02:51:00.880894 3042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc04216457194a4cb93feb5d9acc54ef" candidate_uuid: "30a374a29e454d9f9ef138d36e35e151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9266ccbe346fc9d20626d3129cbfa"
I20260806 02:51:00.880983 2664 raft_consensus.cc:3037] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:00.881050 3042 raft_consensus.cc:3037] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:00.882202 3042 raft_consensus.cc:2454] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a374a29e454d9f9ef138d36e35e151 in term 1.
I20260806 02:51:00.882220 2664 raft_consensus.cc:2454] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a374a29e454d9f9ef138d36e35e151 in term 1.
I20260806 02:51:00.882531 2758 leader_election.cc:302] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9266ccbe346fc9d20626d3129cbfa, 30a374a29e454d9f9ef138d36e35e151; no voters:
I20260806 02:51:00.882773 3180 raft_consensus.cc:2781] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:00.883055 3180 raft_consensus.cc:686] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 1 LEADER]: Becoming Leader. State: Replica: 30a374a29e454d9f9ef138d36e35e151, State: Running, Role: LEADER
I20260806 02:51:00.883483 3180 consensus_queue.cc:237] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:00.887355 1350 catalog_manager.cc:5495] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130). New cstate: current_term: 1 leader_uuid: "30a374a29e454d9f9ef138d36e35e151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } health_report { overall_health: UNKNOWN } } }
I20260806 02:51:00.904640 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.900831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.24.130.130:35157)
0806 02:51:00.900977 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.900981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.900995 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:51:00.901110 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.901112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.901168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.901262 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:00.901507 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.902029 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.903037 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.903268 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.903367 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.903697 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:00.904177 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:00.904331 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.904409 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.904480 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:51:00.911192 3042 raft_consensus.cc:1261] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Refusing update from remote peer 30a374a29e454d9f9ef138d36e35e151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:00.911209 2664 raft_consensus.cc:1261] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Refusing update from remote peer 30a374a29e454d9f9ef138d36e35e151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:00.912050 3180 consensus_queue.cc:1035] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:00.912333 3181 consensus_queue.cc:1035] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:00.922778 3184 mvcc.cc:205] Tried to move back new op lower bound from 7315393171083591680 to 7315393170981625856. Current Snapshot: MvccSnapshot[applied={T|T < 7315393171083591680}]
I20260806 02:51:00.923014 3185 mvcc.cc:205] Tried to move back new op lower bound from 7315393171083591680 to 7315393170981625856. Current Snapshot: MvccSnapshot[applied={T|T < 7315393171083591680}]
I20260806 02:51:00.930842 3186 mvcc.cc:205] Tried to move back new op lower bound from 7315393171083591680 to 7315393170981625856. Current Snapshot: MvccSnapshot[applied={T|T < 7315393171083591680 or (T in {7315393171083591680})}]
I20260806 02:51:00.935932 1352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:00.936173 1352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:00.942087 1352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f26b4ce127e949b59618b96a68c890de: Sending DeleteTablet for 3 replicas of tablet dc04216457194a4cb93feb5d9acc54ef
I20260806 02:51:00.942994 2814 tablet_service.cc:1514] Processing DeleteTablet for tablet dc04216457194a4cb93feb5d9acc54ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:00 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:35536
I20260806 02:51:00.943641 3197 tablet_replica.cc:330] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: stopping tablet replica
I20260806 02:51:00.943650 2635 tablet_service.cc:1514] Processing DeleteTablet for tablet dc04216457194a4cb93feb5d9acc54ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:00 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:36302
I20260806 02:51:00.944087 3011 tablet_service.cc:1514] Processing DeleteTablet for tablet dc04216457194a4cb93feb5d9acc54ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:00 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:51340
I20260806 02:51:00.944185 3197 raft_consensus.cc:2229] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:51:00.944345 3198 tablet_replica.cc:330] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: stopping tablet replica
I20260806 02:51:00.944705 3198 raft_consensus.cc:2229] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:00.944923 3197 raft_consensus.cc:2258] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:00.945132 3198 raft_consensus.cc:2258] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:00.947173 3198 ts_tablet_manager.cc:1905] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:00.948591 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 1303
I20260806 02:51:00.954118 3198 ts_tablet_manager.cc:1918] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:00.954308 3198 log.cc:1196] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/dc04216457194a4cb93feb5d9acc54ef
I20260806 02:51:00.954933 3198 ts_tablet_manager.cc:1939] T dc04216457194a4cb93feb5d9acc54ef P 3503ae3c442040b4af4744a432245e77: Deleting consensus metadata
I20260806 02:51:00.956750 3200 tablet_replica.cc:330] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: stopping tablet replica
I20260806 02:51:00.957211 3200 raft_consensus.cc:2229] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:00.957645 3200 raft_consensus.cc:2258] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:00.959770 3197 ts_tablet_manager.cc:1905] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:00.959769 3200 ts_tablet_manager.cc:1905] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:00.962944 3200 ts_tablet_manager.cc:1918] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:00.962944 3197 ts_tablet_manager.cc:1918] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:00.963086 3200 log.cc:1196] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/dc04216457194a4cb93feb5d9acc54ef
I20260806 02:51:00.963490 3200 ts_tablet_manager.cc:1939] T dc04216457194a4cb93feb5d9acc54ef P 17b9266ccbe346fc9d20626d3129cbfa: Deleting consensus metadata
I20260806 02:51:00.965179 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.947326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.24.130.189:39919)
0806 02:51:00.947485 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.947489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.947506 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:00.947556 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.947559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.947614 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.947707 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.951167 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.952083 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.953461 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.953912 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.954013 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.960474 (+ 6461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.960479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.960482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.960524 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.962500 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.963418 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.963422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.963424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.963536 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.963964 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.963968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.963970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.964263 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.964390 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.964838 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.965004 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:51:00.968871 2943 connection.cc:535] server connection from 127.0.0.1:51340 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 02:51:00.968914 2571 connection.cc:535] server connection from 127.0.0.1:36302 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:51:00.969347 3197 log.cc:1196] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/dc04216457194a4cb93feb5d9acc54ef
I20260806 02:51:00.970041 3197 ts_tablet_manager.cc:1939] T dc04216457194a4cb93feb5d9acc54ef P 30a374a29e454d9f9ef138d36e35e151: Deleting consensus metadata
W20260806 02:51:00.971711 2755 connection.cc:535] server connection from 127.0.0.1:35536 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:51:00.977996 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.947475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.24.130.188:38599)
0806 02:51:00.947607 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.947612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.967205 (+ 19593us) server_negotiation.cc:407] Connection header received
0806 02:51:00.967312 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.967315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.967374 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.967464 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.968497 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.969389 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.970768 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.971155 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.971288 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.973257 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.973269 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.973272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.973305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.975351 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.975931 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.975936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.975939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.976052 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.976674 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.976678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.976680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.977083 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.977194 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.977627 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.977797 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:51:00.986330 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.976253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.24.130.189:39919)
0806 02:51:00.976414 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.976418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.976863 (+ 445us) server_negotiation.cc:407] Connection header received
0806 02:51:00.977046 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.977049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.977101 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.977156 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.978165 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.978978 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.979972 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.980239 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.980365 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.982192 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.982196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.982199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.982231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.984444 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.984972 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.984975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.984977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.985076 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.985386 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.985389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.985391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.985608 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.985749 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.986010 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.986136 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:51:00.988276 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.976035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.24.130.188:38599)
0806 02:51:00.976168 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.976173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.976672 (+ 499us) server_negotiation.cc:407] Connection header received
0806 02:51:00.976854 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.976858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.976906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.976958 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.979134 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.979915 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.981149 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.981532 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.981673 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.983554 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.983558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.983563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:00.983598 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.985876 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.986734 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.986738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.986740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.986836 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:00.987171 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:00.987175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:00.987176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:00.987440 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.987552 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.987953 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.988068 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:51:01.925755 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:01.925229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:60903)
0806 02:51:01.925531 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:01.925634 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
W20260806 02:51:01.926213 2725 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:44675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:01.930420 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:01.929862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:56841)
0806 02:51:01.930155 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:01.930252 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
W20260806 02:51:01.930981 2903 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:44675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:01.931351 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:01.930922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:49425)
0806 02:51:01.931168 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:01.931273 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
W20260806 02:51:01.931757 3110 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:44675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:02.442122 3215 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f26b4ce127e949b59618b96a68c890de)
I20260806 02:51:02.442309 3216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f26b4ce127e949b59618b96a68c890de)
I20260806 02:51:02.442463 3215 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:02.442553 3216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:02.443594 3215 leader_election.cc:288] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675), 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188:38599)
I20260806 02:51:02.443838 3216 leader_election.cc:288] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675), cc865560c44b4066b5a4d064e91f878f (127.24.130.189:39919)
I20260806 02:51:02.444665 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.443808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43144)
0806 02:51:02.444453 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.444557 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":101,"threads_started":1}
I20260806 02:51:02.444702 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.444266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43150)
0806 02:51:02.444556 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.444615 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":60,"threads_started":1}
W20260806 02:51:02.444882 2459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
W20260806 02:51:02.444881 2178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:02.445931 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.445678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43156)
0806 02:51:02.445792 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.445857 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:51:02.446070 2178 leader_election.cc:334] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:02.446254 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.445930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43162)
0806 02:51:02.446132 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.446185 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260806 02:51:02.446357 2459 leader_election.cc:334] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:02.448472 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.444197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.24.130.189:39919)
0806 02:51:02.444449 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:02.444454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:02.444638 (+ 184us) server_negotiation.cc:407] Connection header received
0806 02:51:02.444827 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:02.444831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:02.444877 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:02.444937 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:02.445372 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.446132 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.447416 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.447846 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.447983 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:02.448038 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:02.448275 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 02:51:02.448348 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":180,"thread_start_us":51,"threads_started":1}
I20260806 02:51:02.448521 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.443808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:39919 (local address 127.0.0.1:49884)
0806 02:51:02.444560 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.444577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:02.444701 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:02.445096 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:02.445127 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:02.445146 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:02.445249 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:02.445258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.446292 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.446295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:02.447281 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:02.447292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.447979 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.447983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:02.448087 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:02.448157 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 02:51:02.448258 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:51:02.448341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":174,"spinlock_wait_cycles":896,"thread_start_us":45,"threads_started":1}
I20260806 02:51:02.448555 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.444077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.24.130.188:38599)
0806 02:51:02.444287 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:02.444292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:02.444313 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:51:02.444481 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:02.444484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:02.444548 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:02.444620 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:02.445311 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.446285 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.447529 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.447971 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.448169 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:02.448220 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:02.448334 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:51:02.448405 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260806 02:51:02.448634 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.443844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:38599 (local address 127.0.0.1:42908)
0806 02:51:02.444189 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.444207 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:02.444320 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:02.444942 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:02.444945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:02.444957 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:02.445175 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:02.445182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.446482 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.446485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:02.447384 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:02.447395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:02.448099 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:02.448105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:02.448210 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:02.448238 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:51:02.448318 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:51:02.448404 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260806 02:51:02.449214 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc865560c44b4066b5a4d064e91f878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5101aa9224954066bca7be6cb8e9b1d0" is_pre_election: true
I20260806 02:51:02.449215 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5101aa9224954066bca7be6cb8e9b1d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cc865560c44b4066b5a4d064e91f878f" is_pre_election: true
I20260806 02:51:02.449446 2201 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5101aa9224954066bca7be6cb8e9b1d0 in term 1.
I20260806 02:51:02.449447 2478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc865560c44b4066b5a4d064e91f878f in term 1.
I20260806 02:51:02.449836 2460 leader_election.cc:302] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5101aa9224954066bca7be6cb8e9b1d0, cc865560c44b4066b5a4d064e91f878f; no voters: f26b4ce127e949b59618b96a68c890de
I20260806 02:51:02.449844 2174 leader_election.cc:302] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5101aa9224954066bca7be6cb8e9b1d0, cc865560c44b4066b5a4d064e91f878f; no voters: f26b4ce127e949b59618b96a68c890de
I20260806 02:51:02.450068 3216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:51:02.450130 3216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f26b4ce127e949b59618b96a68c890de)
I20260806 02:51:02.450171 3216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:51:02.450314 3215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:51:02.450395 3215 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f26b4ce127e949b59618b96a68c890de)
I20260806 02:51:02.450471 3215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:51:02.451558 3216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:02.451658 3215 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:02.452039 3216 leader_election.cc:288] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [CANDIDATE]: Term 2 election: Requested vote from peers f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675), cc865560c44b4066b5a4d064e91f878f (127.24.130.189:39919)
I20260806 02:51:02.452128 3215 leader_election.cc:288] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 2 election: Requested vote from peers f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675), 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188:38599)
I20260806 02:51:02.452353 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5101aa9224954066bca7be6cb8e9b1d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cc865560c44b4066b5a4d064e91f878f"
I20260806 02:51:02.452526 2201 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5101aa9224954066bca7be6cb8e9b1d0 in current term 2: Already voted for candidate cc865560c44b4066b5a4d064e91f878f in this term.
I20260806 02:51:02.452752 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.452413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43174)
0806 02:51:02.452616 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.452679 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:51:02.452837 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.452172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43166)
0806 02:51:02.452698 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.452767 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 02:51:02.452834 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc865560c44b4066b5a4d064e91f878f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5101aa9224954066bca7be6cb8e9b1d0"
I20260806 02:51:02.453006 2478 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cc865560c44b4066b5a4d064e91f878f in current term 2: Already voted for candidate 5101aa9224954066bca7be6cb8e9b1d0 in this term.
I20260806 02:51:02.453562 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.453352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43182)
0806 02:51:02.453463 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.453509 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:51:02.453677 2459 leader_election.cc:334] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:02.453775 2459 leader_election.cc:302] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5101aa9224954066bca7be6cb8e9b1d0; no voters: cc865560c44b4066b5a4d064e91f878f, f26b4ce127e949b59618b96a68c890de
I20260806 02:51:02.453864 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.453541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43184)
0806 02:51:02.453726 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.453774 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 02:51:02.453922 3216 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 02:51:02.454021 2178 leader_election.cc:334] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:02.454093 2178 leader_election.cc:302] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc865560c44b4066b5a4d064e91f878f; no voters: 5101aa9224954066bca7be6cb8e9b1d0, f26b4ce127e949b59618b96a68c890de
I20260806 02:51:02.454239 3215 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 02:51:02.927783 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.927039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:49295)
0806 02:51:02.927566 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.927650 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":125,"threads_started":1}
I20260806 02:51:02.932114 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.931571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:60967)
0806 02:51:02.931906 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.931991 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":135,"threads_started":1}
I20260806 02:51:02.932812 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.932346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:48539)
0806 02:51:02.932616 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:02.932690 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260806 02:51:03.929262 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:03.928662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:44759)
0806 02:51:03.928979 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:03.929076 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":113,"threads_started":1}
I20260806 02:51:03.933611 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:03.932865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:54231)
0806 02:51:03.933404 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:03.933492 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":87,"threads_started":1}
I20260806 02:51:03.933995 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:03.933549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:58903)
0806 02:51:03.933820 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:03.933893 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260806 02:51:04.407078 3238 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:04.407299 3238 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:04.407801 3238 leader_election.cc:288] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675), 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188:38599)
I20260806 02:51:04.408370 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc865560c44b4066b5a4d064e91f878f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5101aa9224954066bca7be6cb8e9b1d0" is_pre_election: true
I20260806 02:51:04.408552 2478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc865560c44b4066b5a4d064e91f878f in term 2.
I20260806 02:51:04.408595 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.408123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43454)
0806 02:51:04.408399 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.408462 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":114,"threads_started":1}
I20260806 02:51:04.408953 2174 leader_election.cc:302] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5101aa9224954066bca7be6cb8e9b1d0, cc865560c44b4066b5a4d064e91f878f; no voters:
I20260806 02:51:04.409176 3238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 02:51:04.409230 3238 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:04.409258 3238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:51:04.409570 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.409330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43468)
0806 02:51:04.409451 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.409513 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:51:04.409683 2178 leader_election.cc:334] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:04.410521 3238 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:04.410924 3238 leader_election.cc:288] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 3 election: Requested vote from peers f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675), 5101aa9224954066bca7be6cb8e9b1d0 (127.24.130.188:38599)
I20260806 02:51:04.411320 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.411078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43484)
0806 02:51:04.411200 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.411253 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:51:04.411352 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc865560c44b4066b5a4d064e91f878f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5101aa9224954066bca7be6cb8e9b1d0"
I20260806 02:51:04.411489 2478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:51:04.412036 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.411815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43500)
0806 02:51:04.411928 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.411980 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:51:04.412156 2178 leader_election.cc:334] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
I20260806 02:51:04.412784 2478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc865560c44b4066b5a4d064e91f878f in term 3.
I20260806 02:51:04.413112 2174 leader_election.cc:302] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5101aa9224954066bca7be6cb8e9b1d0, cc865560c44b4066b5a4d064e91f878f; no voters: f26b4ce127e949b59618b96a68c890de
I20260806 02:51:04.413306 3238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:51:04.413512 3238 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [term 3 LEADER]: Becoming Leader. State: Replica: cc865560c44b4066b5a4d064e91f878f, State: Running, Role: LEADER
I20260806 02:51:04.413810 3238 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } }
I20260806 02:51:04.414995 3241 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc865560c44b4066b5a4d064e91f878f. Latest consensus state: current_term: 3 leader_uuid: "cc865560c44b4066b5a4d064e91f878f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:04.415108 3241 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:04.415455 3243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:04.417698 3243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:04.418140 3243 catalog_manager.cc:1259] Loaded cluster ID: 203476151937425f84de8a39b4accf79
I20260806 02:51:04.418186 3243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:04.418823 3243 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:04.419219 3243 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Loaded TSK: 0
I20260806 02:51:04.420047 3243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:04.460041 2195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:04.463424 2195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:04.467125 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.466811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43524)
0806 02:51:04.466956 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.467035 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 02:51:04.467203 2478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [term 3 FOLLOWER]: Refusing update from remote peer cc865560c44b4066b5a4d064e91f878f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 02:51:04.467742 3238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 02:51:04.467926 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.467668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.0.0.1:43538)
0806 02:51:04.467779 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.467839 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:51:04.468042 2178 consensus_peers.cc:487] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f -> Peer f26b4ce127e949b59618b96a68c890de (127.24.130.190:44675): Couldn't send request to peer f26b4ce127e949b59618b96a68c890de. Status: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:51:04.469359 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc865560c44b4066b5a4d064e91f878f. Latest consensus state: current_term: 3 leader_uuid: "cc865560c44b4066b5a4d064e91f878f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:04.469528 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:04.470770 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cc865560c44b4066b5a4d064e91f878f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:04.470913 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:04.471174 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cc865560c44b4066b5a4d064e91f878f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f26b4ce127e949b59618b96a68c890de" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44675 } } peers { permanent_uuid: "cc865560c44b4066b5a4d064e91f878f" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 39919 } attrs { promote: false } } peers { permanent_uuid: "5101aa9224954066bca7be6cb8e9b1d0" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 38599 } attrs { promote: false } } }
I20260806 02:51:04.471329 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5101aa9224954066bca7be6cb8e9b1d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:04.483237 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.478470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:35157 (local address 127.0.0.1:35546)
0806 02:51:04.478839 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.478853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:04.478930 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:04.479554 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:04.479557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:04.479574 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:04.479680 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.479686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.480729 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.480733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.481871 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.481882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.482769 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.482773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.482864 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.482886 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:51:04.482986 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:51:04.483068 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":1,"thread_start_us":191,"threads_started":1}
I20260806 02:51:04.483227 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.478607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.24.130.130:35157)
0806 02:51:04.479182 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.479187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.479204 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:04.479262 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.479265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.479324 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.479417 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:04.479813 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.480591 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.482013 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.482605 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.482777 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.482836 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.483003 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.483082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":114,"threads_started":1}
I20260806 02:51:04.483772 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.478339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.24.130.129:42361)
0806 02:51:04.479681 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.479686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.479701 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:04.479759 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.479765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.479815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.479895 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:04.480169 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.480977 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.482469 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.483159 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.483487 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.483545 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.483579 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.483647 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":96,"threads_started":1}
I20260806 02:51:04.483711 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.478190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42361 (local address 127.0.0.1:36304)
0806 02:51:04.478607 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.478625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:04.478766 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:04.479924 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:04.479927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:04.479945 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:04.480037 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.480044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.481102 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.481105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.482319 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.482328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.483300 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.483304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.483406 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.483452 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:51:04.483515 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:51:04.483589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":18}
I20260806 02:51:04.484179 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.479250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:33791 (local address 127.0.0.1:51352)
0806 02:51:04.479534 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.479550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:04.479640 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:04.480333 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:04.480336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:04.480354 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:04.480441 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.480446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.481434 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.481438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.482347 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.482357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.483416 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.483420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.483874 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.483907 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:51:04.483992 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:51:04.484062 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260806 02:51:04.484637 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.479390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.24.130.131:33791)
0806 02:51:04.480081 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.480085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.480095 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:51:04.480169 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.480172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.480225 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.480313 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:04.480561 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.481293 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.482512 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.483158 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.483293 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.483342 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.484417 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.484505 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":84,"threads_started":1}
I20260806 02:51:04.484602 2635 tablet_service.cc:1467] Processing CreateTablet for tablet ce353701f68d4bbfb8db34f36201039c (DEFAULT_TABLE table=test-table [id=03cea7a23e7846e29ee22ced34cef68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:04.484964 2635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce353701f68d4bbfb8db34f36201039c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:04.485172 3011 tablet_service.cc:1467] Processing CreateTablet for tablet ce353701f68d4bbfb8db34f36201039c (DEFAULT_TABLE table=test-table [id=03cea7a23e7846e29ee22ced34cef68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:04.485450 3011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce353701f68d4bbfb8db34f36201039c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:04.486975 2814 tablet_service.cc:1467] Processing CreateTablet for tablet ce353701f68d4bbfb8db34f36201039c (DEFAULT_TABLE table=test-table [id=03cea7a23e7846e29ee22ced34cef68e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:04.487210 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce353701f68d4bbfb8db34f36201039c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:04.487869 3260 tablet_bootstrap.cc:492] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: Bootstrap starting.
I20260806 02:51:04.488097 3262 tablet_bootstrap.cc:492] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Bootstrap starting.
I20260806 02:51:04.488940 3260 tablet_bootstrap.cc:654] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:04.488991 3262 tablet_bootstrap.cc:654] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:04.489961 3267 tablet_bootstrap.cc:492] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Bootstrap starting.
I20260806 02:51:04.490255 3262 tablet_bootstrap.cc:492] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: No bootstrap required, opened a new log
I20260806 02:51:04.490347 3262 ts_tablet_manager.cc:1397] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:51:04.490839 3262 raft_consensus.cc:348] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.490936 3267 tablet_bootstrap.cc:654] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:04.490975 3262 raft_consensus.cc:374] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:04.491015 3262 raft_consensus.cc:729] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9266ccbe346fc9d20626d3129cbfa, State: Initialized, Role: FOLLOWER
I20260806 02:51:04.491164 3262 consensus_queue.cc:260] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.491554 3262 ts_tablet_manager.cc:1428] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:04.492354 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.491917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:45343)
0806 02:51:04.492184 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.492241 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
I20260806 02:51:04.492656 3260 tablet_bootstrap.cc:492] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: No bootstrap required, opened a new log
I20260806 02:51:04.492792 3260 ts_tablet_manager.cc:1397] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:51:04.493250 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.492819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:40505)
0806 02:51:04.493123 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.493178 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260806 02:51:04.493343 3260 raft_consensus.cc:348] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.493479 3260 raft_consensus.cc:374] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:04.493525 3260 raft_consensus.cc:729] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3503ae3c442040b4af4744a432245e77, State: Initialized, Role: FOLLOWER
I20260806 02:51:04.493675 3260 consensus_queue.cc:260] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.494051 3260 ts_tablet_manager.cc:1428] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:04.494793 3267 tablet_bootstrap.cc:492] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: No bootstrap required, opened a new log
I20260806 02:51:04.494964 3267 ts_tablet_manager.cc:1397] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:51:04.495230 3111 heartbeater.cc:503] Master 127.24.130.189:39919 was elected leader, sending a full tablet report...
I20260806 02:51:04.495477 3267 raft_consensus.cc:348] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.495645 3267 raft_consensus.cc:374] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:04.495735 3267 raft_consensus.cc:729] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a374a29e454d9f9ef138d36e35e151, State: Initialized, Role: FOLLOWER
I20260806 02:51:04.495754 2728 heartbeater.cc:503] Master 127.24.130.189:39919 was elected leader, sending a full tablet report...
I20260806 02:51:04.495918 3267 consensus_queue.cc:260] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.496291 3267 ts_tablet_manager.cc:1428] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:04.497421 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.496717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:59837)
0806 02:51:04.497252 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.497319 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":124,"threads_started":1}
I20260806 02:51:04.498256 2904 heartbeater.cc:503] Master 127.24.130.189:39919 was elected leader, sending a full tablet report...
I20260806 02:51:04.498299 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.497908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:34035)
0806 02:51:04.498175 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.498226 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260806 02:51:04.498886 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.494794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:43357)
0806 02:51:04.498709 (+ 3915us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.498774 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3826,"thread_start_us":120,"threads_started":1}
I20260806 02:51:04.499792 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.499362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:49911)
0806 02:51:04.499646 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.499710 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
I20260806 02:51:04.660773 3274 raft_consensus.cc:482] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:04.660976 3274 raft_consensus.cc:504] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.662168 3274 leader_election.cc:288] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157), 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:04.666684 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.662555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:40675 (local address 127.24.130.130:35157)
0806 02:51:04.662723 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.662727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.662777 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:51:04.662981 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.662987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.663040 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.663152 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:04.663593 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.664120 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.665584 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.665991 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.666124 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.666182 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.666437 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.666518 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:51:04.666700 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.662634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:44349 (local address 127.24.130.131:33791)
0806 02:51:04.662817 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.662822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.662839 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:04.663018 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.663021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.663063 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.663169 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:04.663602 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.664120 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.665548 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.665959 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.666105 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.666170 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.666467 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.666543 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 02:51:04.666698 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.662436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:35157 (local address 127.24.130.129:40675)
0806 02:51:04.662684 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.662706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:04.662841 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:04.663174 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:04.663177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:04.663212 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:04.663472 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.663479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.664283 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.664286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.665418 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.665429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.666120 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.666123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.666284 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.666305 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:51:04.666408 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:51:04.666490 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":167,"spinlock_wait_cycles":10368}
I20260806 02:51:04.666714 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.662498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:33791 (local address 127.24.130.129:44349)
0806 02:51:04.662757 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.662772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:04.662874 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:04.663179 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:04.663182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:04.663214 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:04.663481 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.663489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.664249 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.664252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.665420 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:04.665430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.666110 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.666114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:04.666324 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.666343 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:51:04.666449 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:51:04.666526 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":146,"spinlock_wait_cycles":4608}
I20260806 02:51:04.667244 3042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce353701f68d4bbfb8db34f36201039c" candidate_uuid: "3503ae3c442040b4af4744a432245e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9266ccbe346fc9d20626d3129cbfa" is_pre_election: true
I20260806 02:51:04.667425 3042 raft_consensus.cc:2454] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3503ae3c442040b4af4744a432245e77 in term 0.
I20260806 02:51:04.667861 2572 leader_election.cc:302] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9266ccbe346fc9d20626d3129cbfa, 3503ae3c442040b4af4744a432245e77; no voters:
I20260806 02:51:04.667868 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce353701f68d4bbfb8db34f36201039c" candidate_uuid: "3503ae3c442040b4af4744a432245e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a374a29e454d9f9ef138d36e35e151" is_pre_election: true
I20260806 02:51:04.668164 3274 raft_consensus.cc:2781] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:04.668152 2821 raft_consensus.cc:2454] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3503ae3c442040b4af4744a432245e77 in term 0.
I20260806 02:51:04.668236 3274 raft_consensus.cc:482] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:04.668277 3274 raft_consensus.cc:3037] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:04.671979 3274 raft_consensus.cc:504] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.672348 3274 leader_election.cc:288] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [CANDIDATE]: Term 1 election: Requested vote from peers 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157), 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791)
I20260806 02:51:04.672788 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce353701f68d4bbfb8db34f36201039c" candidate_uuid: "3503ae3c442040b4af4744a432245e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a374a29e454d9f9ef138d36e35e151"
I20260806 02:51:04.672863 3042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce353701f68d4bbfb8db34f36201039c" candidate_uuid: "3503ae3c442040b4af4744a432245e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9266ccbe346fc9d20626d3129cbfa"
I20260806 02:51:04.672967 2821 raft_consensus.cc:3037] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:04.673043 3042 raft_consensus.cc:3037] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:04.674160 2821 raft_consensus.cc:2454] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3503ae3c442040b4af4744a432245e77 in term 1.
I20260806 02:51:04.674163 3042 raft_consensus.cc:2454] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3503ae3c442040b4af4744a432245e77 in term 1.
I20260806 02:51:04.674485 2571 leader_election.cc:302] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a374a29e454d9f9ef138d36e35e151, 3503ae3c442040b4af4744a432245e77; no voters:
I20260806 02:51:04.674770 3274 raft_consensus.cc:2781] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:04.675050 3274 raft_consensus.cc:686] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 1 LEADER]: Becoming Leader. State: Replica: 3503ae3c442040b4af4744a432245e77, State: Running, Role: LEADER
I20260806 02:51:04.675354 3274 consensus_queue.cc:237] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } }
I20260806 02:51:04.675909 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.675605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:36829)
0806 02:51:04.675740 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.675810 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:51:04.677948 2195 catalog_manager.cc:5495] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3503ae3c442040b4af4744a432245e77 (127.24.130.129). New cstate: current_term: 1 leader_uuid: "3503ae3c442040b4af4744a432245e77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3503ae3c442040b4af4744a432245e77" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 } health_report { overall_health: UNKNOWN } } }
W20260806 02:51:04.697362 3113 tablet.cc:2369] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:51:04.719262 2908 tablet.cc:2369] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:04.726033 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.722306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.24.130.129:42361)
0806 02:51:04.722484 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.722487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.722502 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:04.722585 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.722588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.722646 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.722776 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:04.723047 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.723634 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.724472 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.724705 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.724796 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.725090 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:04.725595 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:04.725698 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.725773 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.725834 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:51:04.729473 3042 raft_consensus.cc:1261] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Refusing update from remote peer 3503ae3c442040b4af4744a432245e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:04.729734 2821 raft_consensus.cc:1261] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Refusing update from remote peer 3503ae3c442040b4af4744a432245e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:04.730089 3286 consensus_queue.cc:1035] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b9266ccbe346fc9d20626d3129cbfa" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:04.730396 3274 consensus_queue.cc:1035] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a374a29e454d9f9ef138d36e35e151" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:51:04.731817 2729 tablet.cc:2369] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:04.732910 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.732527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:51141)
0806 02:51:04.732713 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.732791 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 02:51:04.733742 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.733327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:34175)
0806 02:51:04.733587 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.733653 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 02:51:04.736467 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.736169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.129:41979)
0806 02:51:04.736313 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.736381 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:51:04.737882 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.737609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.130:37609)
0806 02:51:04.737742 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.737805 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:51:04.740315 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.740026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44675 (local address 127.24.130.131:34805)
0806 02:51:04.740166 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.740225 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:51:04.745901 2195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:04.746120 2195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:04.752894 2195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc865560c44b4066b5a4d064e91f878f: Sending DeleteTablet for 3 replicas of tablet ce353701f68d4bbfb8db34f36201039c
I20260806 02:51:04.753849 2814 tablet_service.cc:1514] Processing DeleteTablet for tablet ce353701f68d4bbfb8db34f36201039c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:04 UTC) from {username='oryx', principal='oryx/127.24.130.189@KRBTEST.COM'} at 127.0.0.1:35546
I20260806 02:51:04.754169 2635 tablet_service.cc:1514] Processing DeleteTablet for tablet ce353701f68d4bbfb8db34f36201039c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:04 UTC) from {username='oryx', principal='oryx/127.24.130.189@KRBTEST.COM'} at 127.0.0.1:36304
I20260806 02:51:04.754566 3011 tablet_service.cc:1514] Processing DeleteTablet for tablet ce353701f68d4bbfb8db34f36201039c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:04 UTC) from {username='oryx', principal='oryx/127.24.130.189@KRBTEST.COM'} at 127.0.0.1:51352
I20260806 02:51:04.754637 3312 tablet_replica.cc:330] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: stopping tablet replica
I20260806 02:51:04.754797 3312 raft_consensus.cc:2229] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:51:04.755129 3312 raft_consensus.cc:2258] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:04.756273 3312 ts_tablet_manager.cc:1905] T ce353701f68d4bbfb8db34f36201039c P 3503ae3c442040b4af4744a432245e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:04.756870 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 2413
I20260806 02:51:04.760705 3314 tablet_replica.cc:330] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: stopping tablet replica
I20260806 02:51:04.760913 3314 raft_consensus.cc:2229] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:04.761101 3314 raft_consensus.cc:2258] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:04.762168 3314 ts_tablet_manager.cc:1905] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:04.763778 3311 tablet_replica.cc:330] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: stopping tablet replica
I20260806 02:51:04.763960 3311 raft_consensus.cc:2229] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:04.764187 3311 raft_consensus.cc:2258] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:04.765031 3311 ts_tablet_manager.cc:1905] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:04.766120 3314 ts_tablet_manager.cc:1918] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:04.766258 3314 log.cc:1196] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/ce353701f68d4bbfb8db34f36201039c
I20260806 02:51:04.766681 3314 ts_tablet_manager.cc:1939] T ce353701f68d4bbfb8db34f36201039c P 17b9266ccbe346fc9d20626d3129cbfa: Deleting consensus metadata
I20260806 02:51:04.767513 3311 ts_tablet_manager.cc:1918] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:04.767627 3311 log.cc:1196] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/ce353701f68d4bbfb8db34f36201039c
I20260806 02:51:04.767802 2179 catalog_manager.cc:4841] TS 17b9266ccbe346fc9d20626d3129cbfa (127.24.130.131:33791): tablet ce353701f68d4bbfb8db34f36201039c (table test-table [id=03cea7a23e7846e29ee22ced34cef68e]) successfully deleted
I20260806 02:51:04.767997 3311 ts_tablet_manager.cc:1939] T ce353701f68d4bbfb8db34f36201039c P 30a374a29e454d9f9ef138d36e35e151: Deleting consensus metadata
I20260806 02:51:04.768785 2178 catalog_manager.cc:4841] TS 30a374a29e454d9f9ef138d36e35e151 (127.24.130.130:35157): tablet ce353701f68d4bbfb8db34f36201039c (table test-table [id=03cea7a23e7846e29ee22ced34cef68e]) successfully deleted
I20260806 02:51:04.769271 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 2735
I20260806 02:51:04.769290 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.768957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42361 (local address 127.0.0.1:36320)
0806 02:51:04.769157 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:04.769216 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:42361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260806 02:51:04.774438 2179 catalog_manager.cc:4584] TS 3503ae3c442040b4af4744a432245e77 (127.24.130.129:42361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce353701f68d4bbfb8db34f36201039c: Network error: Client connection negotiation failed: client connection to 127.24.130.129:42361: connect: Connection refused (error 111)
I20260806 02:51:04.802747 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 2916
I20260806 02:51:04.817706 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 2008
I20260806 02:51:04.831945 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 2147
2026-08-06T02:51:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7944 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:04 dist-test-slave-3kk6 krb5kdc[3320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:04 dist-test-slave-3kk6 krb5kdc[3320](info): set up 2 sockets
Aug 06 02:51:04 dist-test-slave-3kk6 krb5kdc[3320](info): commencing operation
krb5kdc: starting...
W20260806 02:51:06.064057 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab.
Entry for principal oryx/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab.
I20260806 02:51:06.202279 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:33627
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab
--principal=oryx/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:33627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:06.322834 3334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:06.323202 3334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:06.323294 3334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:06.327605 3334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:06.327709 3334 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:51:06.327739 3334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:06.327775 3334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:06.327834 3334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:06.327889 3334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:06.333645 3334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:33627
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.190.keytab
--principal=oryx/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:33627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:06.334929 3334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:06.336059 3334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:51:06.345592 3334 server_base.cc:1034] running on GCE node
W20260806 02:51:06.345577 3343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.345579 3341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.345862 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:06.346443 3334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:06.347673 3334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:06.348855 3334 hybrid_clock.cc:648] HybridClock initialized: now 1785984666348837 us; error 38 us; skew 500 ppm
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:06.355508 3334 init.cc:377] Logged in from keytab as oryx/127.24.130.190@KRBTEST.COM (short username oryx)
I20260806 02:51:06.356632 3334 webserver.cc:460] Webserver started at http://127.24.130.190:37177/ using document root <none> and password file <none>
I20260806 02:51:06.357270 3334 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:06.357337 3334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:06.357591 3334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:06.360015 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "f21ff689c40a4824bdd9d7c2c524c7b5"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "66dbd1de3d2295d914bba8ee35e5b94b"
server_key_iv: "cf73821210f0e88929c3cf16304a6daa"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.360680 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "f21ff689c40a4824bdd9d7c2c524c7b5"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "66dbd1de3d2295d914bba8ee35e5b94b"
server_key_iv: "cf73821210f0e88929c3cf16304a6daa"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.364907 3334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:06.367636 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:06.368969 3334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:06.369107 3334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "f21ff689c40a4824bdd9d7c2c524c7b5"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "66dbd1de3d2295d914bba8ee35e5b94b"
server_key_iv: "cf73821210f0e88929c3cf16304a6daa"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.369240 3334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:06.388738 3334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:06.392369 3334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:06.392611 3334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:06.401659 3334 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:33627
I20260806 02:51:06.401682 3444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:33627 every 8 connection(s)
I20260806 02:51:06.402781 3334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:06.405766 3445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:06.411334 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 3334
I20260806 02:51:06.411495 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:51:06.411861 25098 external_mini_cluster.cc:1428] Setting key 4cf1fbf41708bff33e9182c41fcf9361
I20260806 02:51:06.412293 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Bootstrap starting.
I20260806 02:51:06.415297 3445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:06.416138 3445 log.cc:824] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:06.418285 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: No bootstrap required, opened a new log
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:06.422796 3445 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33627 } }
I20260806 02:51:06.423101 3445 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:06.423171 3445 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21ff689c40a4824bdd9d7c2c524c7b5, State: Initialized, Role: FOLLOWER
I20260806 02:51:06.423765 3445 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33627 } }
I20260806 02:51:06.423977 3445 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:06.424058 3445 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:06.424170 3445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:06.425479 3445 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33627 } }
I20260806 02:51:06.425912 3445 leader_election.cc:302] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21ff689c40a4824bdd9d7c2c524c7b5; no voters:
I20260806 02:51:06.426318 3445 leader_election.cc:288] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:06.426434 3452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:06.426715 3452 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [term 1 LEADER]: Becoming Leader. State: Replica: f21ff689c40a4824bdd9d7c2c524c7b5, State: Running, Role: LEADER
I20260806 02:51:06.427143 3452 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33627 } }
I20260806 02:51:06.427863 3445 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:06.428669 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.413517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.24.130.190:33627)
0806 02:51:06.414093 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.414108 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.414148 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:51:06.414947 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.414968 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.415338 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.415762 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.416674 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.417696 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.418638 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.419201 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.419381 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.422396 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.422402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.422419 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:06.422452 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.424791 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.425304 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.425313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.425322 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.425522 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.425901 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.425904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.425905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.426567 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.426812 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.427316 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.427646 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":209,"threads_started":1}
I20260806 02:51:06.429000 3454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33627 } } }
I20260806 02:51:06.429136 3454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:06.429632 3459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21ff689c40a4824bdd9d7c2c524c7b5. Latest consensus state: current_term: 1 leader_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21ff689c40a4824bdd9d7c2c524c7b5" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 33627 } } }
I20260806 02:51:06.429788 3459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5 [sys.catalog]: This master's current role is: LEADER
W20260806 02:51:06.433349 3478 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:51:06.433439 3478 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:51:06.433529 3469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:06.434248 3469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:06.440151 3469 catalog_manager.cc:1347] Generated new cluster ID: 306c378f5280469490fbd7ccce8b511e
I20260806 02:51:06.440306 3469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:06.458561 3469 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:06.460071 3469 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:06.467234 3469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Generated new TSK 0
I20260806 02:51:06.468252 3469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab.
Entry for principal oryx/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab.
I20260806 02:51:06.506536 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:33627
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:06.628512 3488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:06.628931 3488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:06.628985 3488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:06.633297 3488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:06.633384 3488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:06.633427 3488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:51:06.633462 3488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:06.639668 3488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:33627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:06.640820 3488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:06.641871 3488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:51:06.652369 3488 server_base.cc:1034] running on GCE node
W20260806 02:51:06.652312 3494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.652312 3496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.652320 3493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:06.653008 3488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:06.653728 3488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:06.654922 3488 hybrid_clock.cc:648] HybridClock initialized: now 1785984666654891 us; error 48 us; skew 500 ppm
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:06.657871 3488 init.cc:377] Logged in from keytab as oryx/127.24.130.129@KRBTEST.COM (short username oryx)
I20260806 02:51:06.658918 3488 webserver.cc:460] Webserver started at http://127.24.130.129:45951/ using document root <none> and password file <none>
I20260806 02:51:06.659538 3488 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:06.659637 3488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:06.659898 3488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:06.662127 3488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "eee208549fa54101badd9a03a4e6a88c"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "f681d9253d90b95e5d93df5a2418d42d"
server_key_iv: "3e4a087fe4d9a346007d6cbabba45b02"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.662818 3488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "eee208549fa54101badd9a03a4e6a88c"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "f681d9253d90b95e5d93df5a2418d42d"
server_key_iv: "3e4a087fe4d9a346007d6cbabba45b02"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.666944 3488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:51:06.669544 3505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:06.670894 3488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:06.671037 3488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "eee208549fa54101badd9a03a4e6a88c"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "f681d9253d90b95e5d93df5a2418d42d"
server_key_iv: "3e4a087fe4d9a346007d6cbabba45b02"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.671166 3488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:06.711098 3488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:06.714452 3488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:06.714718 3488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:06.716329 3488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:06.716425 3488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:06.716503 3488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:06.716552 3488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:06.728457 3488 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:33915
I20260806 02:51:06.728578 3673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:33915 every 8 connection(s)
I20260806 02:51:06.729561 3488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:06.732734 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.717576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:53335 (local address 127.24.130.190:33627)
0806 02:51:06.717751 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.717754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.718984 (+ 1230us) server_negotiation.cc:407] Connection header received
0806 02:51:06.720256 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.720260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.720329 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.720442 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.722274 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.723059 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.725004 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.725277 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.725444 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.728830 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.728835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.728838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:06.728871 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.730841 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.731351 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.731354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.731355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.731434 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.731753 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.731755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.731757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.731943 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.732022 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.732488 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.732599 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:51:06.733536 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.718123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.24.130.129:53335)
0806 02:51:06.718811 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:06.718862 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:06.720010 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:06.720683 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:06.720694 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:06.721252 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:06.722060 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.722085 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.723182 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.723186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.724833 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.724847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.725518 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.725525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.725753 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.726608 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:06.726628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:06.728613 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:06.730961 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.730969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.730984 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.731257 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.731519 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.731521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.731523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.731655 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.732034 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:06.732039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:06.732266 (+ 227us) client_negotiation.cc:769] Sending connection context
0806 02:51:06.732432 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:51:06.732686 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":164,"threads_started":1}
I20260806 02:51:06.735107 3674 heartbeater.cc:347] Connected to a master server at 127.24.130.190:33627
I20260806 02:51:06.735286 3674 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:06.735776 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 3488
I20260806 02:51:06.735914 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:06.735966 3674 heartbeater.cc:511] Master 127.24.130.190:33627 requested a full tablet report, sending...
I20260806 02:51:06.736272 25098 external_mini_cluster.cc:1428] Setting key dcabf30f17ba937477b9f5700e32fe07
I20260806 02:51:06.737811 3378 ts_manager.cc:194] Registered new tserver with Master: eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915)
I20260806 02:51:06.739558 3378 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:53335
I20260806 02:51:06.745757 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.737482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.24.130.129:56225)
0806 02:51:06.737837 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:06.737854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:06.738002 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:06.738310 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:06.738313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:06.738540 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:06.738642 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.738700 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.739477 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.739480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.740146 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.740183 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.740861 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.740863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.740937 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.741756 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:06.741768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:06.742268 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:06.744121 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.744125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.744128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.744351 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.744666 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.744680 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.744682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.744777 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.745289 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:06.745291 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:06.745380 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:51:06.745475 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:51:06.745596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":6,"thread_start_us":139,"threads_started":1}
I20260806 02:51:06.745983 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.737572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:56225 (local address 127.24.130.190:33627)
0806 02:51:06.737696 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.737701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.737924 (+ 223us) server_negotiation.cc:407] Connection header received
0806 02:51:06.738137 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.738140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.738200 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.738292 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.738798 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.739344 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.740330 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.740649 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.740724 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.742413 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.742416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.742419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:06.742445 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.743998 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.744466 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.744469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.744471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.744564 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.744879 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.744883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.744885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.745163 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.745295 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.745579 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.745731 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
WARNING: no policy specified for oryx/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab.
Entry for principal oryx/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab.
I20260806 02:51:06.769686 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:33627
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:06.890898 3687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:06.891258 3687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:06.891347 3687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:06.895553 3687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:06.895656 3687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:06.895733 3687 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:51:06.895824 3687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:06.901345 3687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:33627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:06.902520 3687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:06.903657 3687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:06.913764 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:06.913872 3687 server_base.cc:1034] running on GCE node
W20260806 02:51:06.913774 3696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.913776 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:06.914546 3687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:06.915282 3687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:06.916481 3687 hybrid_clock.cc:648] HybridClock initialized: now 1785984666916455 us; error 40 us; skew 500 ppm
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:06.919677 3687 init.cc:377] Logged in from keytab as oryx/127.24.130.130@KRBTEST.COM (short username oryx)
I20260806 02:51:06.920714 3687 webserver.cc:460] Webserver started at http://127.24.130.130:38037/ using document root <none> and password file <none>
I20260806 02:51:06.921553 3687 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:06.921664 3687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:06.921962 3687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:06.924278 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "1117ae42c447474c991fdb7368235fdc"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "0dd948d345b679b268f6b23e93b00575"
server_key_iv: "3204691de01affddfe94db4393c033a2"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.924919 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "1117ae42c447474c991fdb7368235fdc"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "0dd948d345b679b268f6b23e93b00575"
server_key_iv: "3204691de01affddfe94db4393c033a2"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.929199 3687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:06.932235 3704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:06.933601 3687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:06.933715 3687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "1117ae42c447474c991fdb7368235fdc"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-3kk6"
server_key: "0dd948d345b679b268f6b23e93b00575"
server_key_iv: "3204691de01affddfe94db4393c033a2"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.933905 3687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:06.946197 3687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:06.949668 3687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:06.949920 3687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:06.951649 3687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:06.951726 3687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:06.951771 3687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:06.951828 3687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:51:06 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:06.963883 3687 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:43081
I20260806 02:51:06.963958 3885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:43081 every 8 connection(s)
I20260806 02:51:06.965046 3687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:06.967170 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.952480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:49263 (local address 127.24.130.190:33627)
0806 02:51:06.952621 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.952625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.953764 (+ 1139us) server_negotiation.cc:407] Connection header received
0806 02:51:06.954974 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.954978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.955050 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.955113 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.956785 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.957713 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.959119 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.959420 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.959544 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.962959 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.962963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.962965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:06.963003 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.964666 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.965339 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.965342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.965343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.965461 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.965900 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.965904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.965906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.966148 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.966272 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.966817 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.966992 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:51:06.968102 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.953050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.24.130.130:49263)
0806 02:51:06.953590 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:06.953636 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:06.954613 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:06.955610 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:06.955617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:06.956007 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:06.956568 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.956590 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.957861 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.957864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.958969 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.958981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.959617 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.959625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.959892 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.960697 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:06.960722 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:06.962726 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:06.964809 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.964816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.964831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.965220 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.965589 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.965593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.965596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.965775 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.966268 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:06.966275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:06.966532 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 02:51:06.966755 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 02:51:06.967103 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":166,"threads_started":1}
I20260806 02:51:06.968109 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 3687
I20260806 02:51:06.968250 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:06.968583 25098 external_mini_cluster.cc:1428] Setting key 27f362f96f9c539842dc9814b99a2f5f
I20260806 02:51:06.969746 3886 heartbeater.cc:347] Connected to a master server at 127.24.130.190:33627
I20260806 02:51:06.969897 3886 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:06.970534 3886 heartbeater.cc:511] Master 127.24.130.190:33627 requested a full tablet report, sending...
I20260806 02:51:06.971743 3378 ts_manager.cc:194] Registered new tserver with Master: 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081)
I20260806 02:51:06.972357 3378 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:49263
WARNING: no policy specified for oryx/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:51:06.981051 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.972637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:50303 (local address 127.24.130.190:33627)
0806 02:51:06.972774 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.972779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.973113 (+ 334us) server_negotiation.cc:407] Connection header received
0806 02:51:06.973249 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.973252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.973301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.973393 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.974029 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.974823 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.975831 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.976134 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.976242 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.977483 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.977486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.977489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:06.977514 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.979125 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.979581 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.979585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.979587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.979679 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:06.979978 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:06.979981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:06.979983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:06.980252 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.980365 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.980731 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.980849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:51:06.981046 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.972647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.24.130.130:50303)
0806 02:51:06.973010 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:06.973029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:06.973141 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:06.973433 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:06.973438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:06.973696 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:06.973900 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.973911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.974986 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.974988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.975676 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:06.975692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.976264 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.976268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:06.976380 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.976975 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:06.976987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:06.977334 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:06.979248 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.979252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.979254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.979473 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.979775 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:06.979779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:06.979781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:06.979873 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:06.980398 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:06.980402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:06.980523 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 02:51:06.980664 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:51:06.980841 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab.
Entry for principal oryx/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab.
I20260806 02:51:06.999708 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:33627
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:07.120038 3897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:07.120335 3897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:07.120383 3897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:07.124482 3897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:07.124555 3897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:07.124598 3897 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:51:07.124678 3897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:07.130306 3897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.130.190:33627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:07.131502 3897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:07.132628 3897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:07.143033 3906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:07.143041 3908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:07.143007 3905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:07.143293 3897 server_base.cc:1034] running on GCE node
I20260806 02:51:07.143864 3897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:07.144573 3897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:07.145917 3897 hybrid_clock.cc:648] HybridClock initialized: now 1785984667145750 us; error 38 us; skew 500 ppm
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984667, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:07.149099 3897 init.cc:377] Logged in from keytab as oryx/127.24.130.131@KRBTEST.COM (short username oryx)
I20260806 02:51:07.149986 3897 webserver.cc:460] Webserver started at http://127.24.130.131:41013/ using document root <none> and password file <none>
I20260806 02:51:07.150523 3897 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:07.150585 3897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:07.150840 3897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:07.152956 3897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "c815aefee1744205a5197698c6db1272"
format_stamp: "Formatted at 2026-08-06 02:51:07 on dist-test-slave-3kk6"
server_key: "3fb0cfefb1647a9bd486c2dcc37a6790"
server_key_iv: "f2d6dc94dff2d962439c57a4d33082c0"
server_key_version: "encryptionkey@0"
I20260806 02:51:07.153512 3897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "c815aefee1744205a5197698c6db1272"
format_stamp: "Formatted at 2026-08-06 02:51:07 on dist-test-slave-3kk6"
server_key: "3fb0cfefb1647a9bd486c2dcc37a6790"
server_key_iv: "f2d6dc94dff2d962439c57a4d33082c0"
server_key_version: "encryptionkey@0"
I20260806 02:51:07.157501 3897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:07.160574 3919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:07.162012 3897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:07.162154 3897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "c815aefee1744205a5197698c6db1272"
format_stamp: "Formatted at 2026-08-06 02:51:07 on dist-test-slave-3kk6"
server_key: "3fb0cfefb1647a9bd486c2dcc37a6790"
server_key_iv: "f2d6dc94dff2d962439c57a4d33082c0"
server_key_version: "encryptionkey@0"
I20260806 02:51:07.162285 3897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:07.204406 3897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:07.208227 3897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:07.208469 3897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:07.210049 3897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:07.210103 3897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:07.210170 3897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:07.210203 3897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984667, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:07.221948 3897 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:36025
I20260806 02:51:07.222024 4091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:36025 every 8 connection(s)
I20260806 02:51:07.223184 3897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:07.225103 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.211135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:43943 (local address 127.24.130.190:33627)
0806 02:51:07.211277 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.211281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.212418 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 02:51:07.213535 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.213539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.213604 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.213712 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.215480 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.216374 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.217678 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.217933 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.218057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.220964 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:07.220968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:07.220971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:07.221003 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:07.222984 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.223555 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.223559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.223561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.223672 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.224112 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.224121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.224123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.224329 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:07.224410 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.224871 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.224978 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:51:07.225970 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.211741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.24.130.131:43943)
0806 02:51:07.212250 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.212294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.213258 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.213970 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.213982 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.214482 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:07.215276 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.215310 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.216542 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.216547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.217546 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.217556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.218114 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.218118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.218348 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.219098 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:07.219114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:07.220763 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:07.223104 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:07.223109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:07.223121 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:07.223449 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:07.223794 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:07.223797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:07.223799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:07.223989 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:07.224430 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:07.224435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:07.224660 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.224819 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.225076 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":151,"threads_started":1}
I20260806 02:51:07.227252 4092 heartbeater.cc:347] Connected to a master server at 127.24.130.190:33627
I20260806 02:51:07.227430 4092 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:07.228036 4092 heartbeater.cc:511] Master 127.24.130.190:33627 requested a full tablet report, sending...
I20260806 02:51:07.229199 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 3897
I20260806 02:51:07.229241 3379 ts_manager.cc:194] Registered new tserver with Master: c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:07.229369 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:07.229727 25098 external_mini_cluster.cc:1428] Setting key 159ae5c59b4e50b1feace8f6e9504dba
I20260806 02:51:07.230042 3379 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:43943
I20260806 02:51:07.231920 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:51:07.238101 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.229868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:36855 (local address 127.24.130.190:33627)
0806 02:51:07.229995 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.229999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.230233 (+ 234us) server_negotiation.cc:407] Connection header received
0806 02:51:07.230389 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.230392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.230441 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.230516 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.231152 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.231886 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.232868 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.233260 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.233386 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.234545 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:07.234548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:07.234551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:07.234576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:07.236311 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.236782 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.236786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.236788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.236885 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.237185 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.237189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.237191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.237390 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:07.237499 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.237808 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.237927 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:51:07.238112 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.229811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.24.130.131:36855)
0806 02:51:07.230150 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.230168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.230301 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.230632 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.230635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.230913 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:07.231051 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.231062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.232028 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.232032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.232734 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.232744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.233378 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.233380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.233449 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.234023 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:07.234038 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:07.234404 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:07.236438 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:07.236444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:07.236448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:07.236672 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:07.236986 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:07.236989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:07.236991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:07.237086 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:07.237504 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:07.237508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:07.237613 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.237740 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.237931 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:51:07.259837 3448 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:07.253221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.24.130.190:33627)
0806 02:51:07.253376 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.253380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.253394 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:51:07.253439 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.253442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.253483 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.253564 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.254299 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.254903 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.255730 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.255937 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.256013 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.259583 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54218: BlockingRecv error: recv got EOF from 127.0.0.1:54218 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:07.270089 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.261314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.24.130.190:33627)
0806 02:51:07.261446 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.261449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.261580 (+ 131us) server_negotiation.cc:407] Connection header received
0806 02:51:07.261680 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.261682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.261723 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.261792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.262385 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.262963 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.263751 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.263965 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.264038 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.266877 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:07.266880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:07.266882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:07.266907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:07.268412 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.268818 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.268820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.268822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.268899 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.269190 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.269193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.269194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.269528 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:07.269635 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.269866 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.269964 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:51:07.273196 3378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:07.276542 3378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:07.291816 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.286838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.24.130.130:43081)
0806 02:51:07.287226 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.287235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.287253 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:51:07.287313 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.287316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.287484 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.287633 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.287886 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.288945 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.290221 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.291057 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.291298 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.291435 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.291576 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.291652 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":142,"threads_started":1}
I20260806 02:51:07.291826 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.286184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:43081 (local address 127.0.0.1:38822)
0806 02:51:07.286858 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.286874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.286968 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.287640 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.287643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.287664 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.287749 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.287755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.289071 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.289074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.290087 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.290097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.291206 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.291210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.291362 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.291412 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.291528 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.291652 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":104,"thread_start_us":114,"threads_started":1}
I20260806 02:51:07.292346 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.285611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:36025 (local address 127.0.0.1:35052)
0806 02:51:07.286246 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.286263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.286354 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.288041 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.288044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.288062 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.288146 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.288154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.289263 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.289266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.290124 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.290134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.291076 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.291080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.292001 (+ 921us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.292027 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.292106 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.292174 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":29,"thread_start_us":120,"threads_started":1}
I20260806 02:51:07.292774 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.286753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.24.130.131:36025)
0806 02:51:07.287618 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.287629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.287650 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:51:07.287717 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.287722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.287911 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.288063 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.288262 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.289134 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.290258 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.290941 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.291115 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.291270 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.292512 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.292616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":135,"threads_started":1}
I20260806 02:51:07.292882 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.285421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33915 (local address 127.0.0.1:48344)
0806 02:51:07.286008 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.286063 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.286216 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.287205 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.287210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.287230 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.287342 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.287347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.288957 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.288960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.291246 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.291255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.292492 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.292495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.292581 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.292602 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.292675 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.292735 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":834,"thread_start_us":111,"threads_started":1}
I20260806 02:51:07.293284 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.285611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.24.130.129:33915)
0806 02:51:07.286524 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.286532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.286557 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:51:07.286636 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.286643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.286891 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.287042 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.287466 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.288663 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.291356 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.292180 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.292297 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.292422 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.293045 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.293154 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":117,"threads_started":1}
I20260806 02:51:07.295367 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 (DEFAULT_TABLE table=test-table [id=6145576b790948d79eb96b85448943c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.295924 3988 tablet_service.cc:1467] Processing CreateTablet for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 (DEFAULT_TABLE table=test-table [id=6145576b790948d79eb96b85448943c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.295888 3574 tablet_service.cc:1467] Processing CreateTablet for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 (DEFAULT_TABLE table=test-table [id=6145576b790948d79eb96b85448943c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.296437 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.296878 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.296886 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.303328 4136 tablet_bootstrap.cc:492] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Bootstrap starting.
I20260806 02:51:07.304767 4137 tablet_bootstrap.cc:492] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Bootstrap starting.
I20260806 02:51:07.305289 4139 tablet_bootstrap.cc:492] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Bootstrap starting.
I20260806 02:51:07.306077 4136 tablet_bootstrap.cc:654] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.306928 4137 tablet_bootstrap.cc:654] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.307065 4136 log.cc:824] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:07.307502 4139 tablet_bootstrap.cc:654] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.307718 4137 log.cc:824] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:07.308334 4139 log.cc:824] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:07.309199 4136 tablet_bootstrap.cc:492] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: No bootstrap required, opened a new log
I20260806 02:51:07.309399 4136 ts_tablet_manager.cc:1397] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:51:07.309659 4137 tablet_bootstrap.cc:492] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: No bootstrap required, opened a new log
I20260806 02:51:07.309903 4137 ts_tablet_manager.cc:1397] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:51:07.310132 4139 tablet_bootstrap.cc:492] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: No bootstrap required, opened a new log
I20260806 02:51:07.310586 4139 ts_tablet_manager.cc:1397] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:51:07.313338 4139 raft_consensus.cc:348] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.313585 4139 raft_consensus.cc:374] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.313661 4139 raft_consensus.cc:729] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c815aefee1744205a5197698c6db1272, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.313532 4136 raft_consensus.cc:348] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.313802 4136 raft_consensus.cc:374] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.313856 4136 raft_consensus.cc:729] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.314186 4139 consensus_queue.cc:260] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.314263 4136 consensus_queue.cc:260] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.314446 4137 raft_consensus.cc:348] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.314765 4137 raft_consensus.cc:374] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.314870 4137 raft_consensus.cc:729] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee208549fa54101badd9a03a4e6a88c, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.314879 4092 heartbeater.cc:503] Master 127.24.130.190:33627 was elected leader, sending a full tablet report...
I20260806 02:51:07.315212 4139 ts_tablet_manager.cc:1428] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:51:07.315290 4136 ts_tablet_manager.cc:1428] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:51:07.315416 4137 consensus_queue.cc:260] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.315611 3886 heartbeater.cc:503] Master 127.24.130.190:33627 was elected leader, sending a full tablet report...
I20260806 02:51:07.316267 4137 ts_tablet_manager.cc:1428] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:51:07.316573 3674 heartbeater.cc:503] Master 127.24.130.190:33627 was elected leader, sending a full tablet report...
W20260806 02:51:07.466408 3887 tablet.cc:2369] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:07.466776 4144 raft_consensus.cc:482] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:07.467017 4144 raft_consensus.cc:504] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.468262 4144 leader_election.cc:288] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081)
I20260806 02:51:07.472669 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.468765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:40577 (local address 127.24.130.130:43081)
0806 02:51:07.468916 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.468921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.469156 (+ 235us) server_negotiation.cc:407] Connection header received
0806 02:51:07.469311 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.469315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.469383 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.469461 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.469793 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.470369 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.471658 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.472072 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.472218 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.472276 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.472468 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.472538 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:51:07.472669 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.468612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:43081 (local address 127.24.130.131:40577)
0806 02:51:07.469039 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.469057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.469189 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.469487 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.469491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.469507 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.469678 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.469689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.470499 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.470504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.471531 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.471540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.472217 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.472221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.472331 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.472354 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.472451 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.472529 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":25}
I20260806 02:51:07.473126 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.468963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:59507 (local address 127.24.130.129:33915)
0806 02:51:07.469098 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.469103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.469305 (+ 202us) server_negotiation.cc:407] Connection header received
0806 02:51:07.469424 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.469426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.469499 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.469603 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.469900 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.470394 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.471972 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.472511 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.472645 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.472696 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.472921 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.473000 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:51:07.473441 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e95a7f6ec254a28ad53d2cf8b515ca1" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1117ae42c447474c991fdb7368235fdc" is_pre_election: true
I20260806 02:51:07.473594 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.468612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33915 (local address 127.24.130.131:59507)
0806 02:51:07.469166 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.469183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.469323 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.469619 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.469622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.469638 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.469756 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.469760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.470517 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.470522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.471833 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.471843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.472645 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.472649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.472753 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.472786 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.473367 (+ 581us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.473444 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":165,"thread_start_us":170,"threads_started":1}
I20260806 02:51:07.473817 3810 raft_consensus.cc:2454] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 0.
W20260806 02:51:07.474195 4094 tablet.cc:2369] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:07.474181 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e95a7f6ec254a28ad53d2cf8b515ca1" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c" is_pre_election: true
I20260806 02:51:07.474490 3599 raft_consensus.cc:2454] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 0.
I20260806 02:51:07.474478 3924 leader_election.cc:302] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, c815aefee1744205a5197698c6db1272; no voters:
I20260806 02:51:07.474895 4144 raft_consensus.cc:2781] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:07.474970 4144 raft_consensus.cc:482] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:07.475054 4144 raft_consensus.cc:3037] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.476383 4144 raft_consensus.cc:504] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
I20260806 02:51:07.476804 4144 leader_election.cc:288] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 election: Requested vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081)
I20260806 02:51:07.477154 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e95a7f6ec254a28ad53d2cf8b515ca1" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c"
I20260806 02:51:07.477200 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e95a7f6ec254a28ad53d2cf8b515ca1" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1117ae42c447474c991fdb7368235fdc"
I20260806 02:51:07.477273 3599 raft_consensus.cc:3037] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.477355 3810 raft_consensus.cc:3037] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.478361 3599 raft_consensus.cc:2454] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 1.
I20260806 02:51:07.478482 3810 raft_consensus.cc:2454] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 1.
I20260806 02:51:07.478734 3921 leader_election.cc:302] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c815aefee1744205a5197698c6db1272, eee208549fa54101badd9a03a4e6a88c; no voters:
I20260806 02:51:07.478929 4144 raft_consensus.cc:2781] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:07.479162 4144 raft_consensus.cc:686] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 1 LEADER]: Becoming Leader. State: Replica: c815aefee1744205a5197698c6db1272, State: Running, Role: LEADER
I20260806 02:51:07.479544 4144 consensus_queue.cc:237] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } }
W20260806 02:51:07.481629 3675 tablet.cc:2369] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:07.482868 3379 catalog_manager.cc:5495] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 reported cstate change: term changed from 0 to 1, leader changed from <none> to c815aefee1744205a5197698c6db1272 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "c815aefee1744205a5197698c6db1272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } health_report { overall_health: UNKNOWN } } }
I20260806 02:51:07.531527 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.527463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.24.130.131:36025)
0806 02:51:07.527613 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.527618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.527635 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:07.527833 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.527835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.527901 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.528019 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:07.528313 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.528977 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.529916 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.530151 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.530244 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.530530 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.531096 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.531211 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.531259 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.531314 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:51:07.537923 3599 raft_consensus.cc:1261] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Refusing update from remote peer c815aefee1744205a5197698c6db1272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:07.537994 3810 raft_consensus.cc:1261] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Refusing update from remote peer c815aefee1744205a5197698c6db1272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:07.538756 4144 consensus_queue.cc:1035] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:07.539070 4151 consensus_queue.cc:1035] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:07.547216 4156 mvcc.cc:205] Tried to move back new op lower bound from 7315393198226231296 to 7315393197998714880. Current Snapshot: MvccSnapshot[applied={T|T < 7315393198226231296}]
I20260806 02:51:07.556085 3379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:07.556269 3379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:07.559355 3379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Sending DeleteTablet for 3 replicas of tablet 0e95a7f6ec254a28ad53d2cf8b515ca1
I20260806 02:51:07.560254 3988 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:07 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:35052
I20260806 02:51:07.560443 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:07 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:38822
I20260806 02:51:07.560882 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:07 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260806 02:51:07.560951 4173 tablet_replica.cc:330] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: stopping tablet replica
I20260806 02:51:07.561312 4173 raft_consensus.cc:2229] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:51:07.561420 3379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:51:07.561559 4174 tablet_replica.cc:330] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: stopping tablet replica
I20260806 02:51:07.561862 4174 raft_consensus.cc:2229] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 02:51:07.561988 3379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:07.562120 4173 raft_consensus.cc:2258] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:07.562260 4174 raft_consensus.cc:2258] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:07.562999 4176 tablet_replica.cc:330] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: stopping tablet replica
I20260806 02:51:07.563357 4176 raft_consensus.cc:2229] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:07.563721 4176 raft_consensus.cc:2258] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:07.564054 4173 ts_tablet_manager.cc:1905] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:07.565709 4174 ts_tablet_manager.cc:1905] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:07.566507 4176 ts_tablet_manager.cc:1905] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:07.568190 3988 tablet_service.cc:1467] Processing CreateTablet for tablet ec5e89cf0a264fb799289916373f6769 (DEFAULT_TABLE table=test-table [id=ddb77c6aa3bc4a6fafd966ee862d4121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.568257 3574 tablet_service.cc:1467] Processing CreateTablet for tablet ec5e89cf0a264fb799289916373f6769 (DEFAULT_TABLE table=test-table [id=ddb77c6aa3bc4a6fafd966ee862d4121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.568493 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5e89cf0a264fb799289916373f6769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.568493 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5e89cf0a264fb799289916373f6769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.568763 4174 ts_tablet_manager.cc:1918] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:07.568864 4174 log.cc:1196] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/0e95a7f6ec254a28ad53d2cf8b515ca1
I20260806 02:51:07.569187 4174 ts_tablet_manager.cc:1939] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P 1117ae42c447474c991fdb7368235fdc: Deleting consensus metadata
I20260806 02:51:07.570923 3785 tablet_service.cc:1467] Processing CreateTablet for tablet ec5e89cf0a264fb799289916373f6769 (DEFAULT_TABLE table=test-table [id=ddb77c6aa3bc4a6fafd966ee862d4121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.571197 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5e89cf0a264fb799289916373f6769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.573094 4173 ts_tablet_manager.cc:1918] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:07.573196 4173 log.cc:1196] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/0e95a7f6ec254a28ad53d2cf8b515ca1
I20260806 02:51:07.573518 4173 ts_tablet_manager.cc:1939] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P c815aefee1744205a5197698c6db1272: Deleting consensus metadata
I20260806 02:51:07.573750 4139 tablet_bootstrap.cc:492] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Bootstrap starting.
I20260806 02:51:07.574311 4176 ts_tablet_manager.cc:1918] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:07.574421 4176 log.cc:1196] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/0e95a7f6ec254a28ad53d2cf8b515ca1
I20260806 02:51:07.574942 4176 ts_tablet_manager.cc:1939] T 0e95a7f6ec254a28ad53d2cf8b515ca1 P eee208549fa54101badd9a03a4e6a88c: Deleting consensus metadata
I20260806 02:51:07.575040 4139 tablet_bootstrap.cc:654] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.575709 4136 tablet_bootstrap.cc:492] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Bootstrap starting.
I20260806 02:51:07.576683 4136 tablet_bootstrap.cc:654] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.577384 4139 tablet_bootstrap.cc:492] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: No bootstrap required, opened a new log
I20260806 02:51:07.577471 4139 ts_tablet_manager.cc:1397] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:51:07.577896 4139 raft_consensus.cc:348] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.577965 4136 tablet_bootstrap.cc:492] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: No bootstrap required, opened a new log
I20260806 02:51:07.577991 4139 raft_consensus.cc:374] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.578013 4139 raft_consensus.cc:729] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c815aefee1744205a5197698c6db1272, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.578045 4136 ts_tablet_manager.cc:1397] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:51:07.578166 4139 consensus_queue.cc:260] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.578460 4139 ts_tablet_manager.cc:1428] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:51:07.578838 4136 raft_consensus.cc:348] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.578956 4136 raft_consensus.cc:374] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.578996 4136 raft_consensus.cc:729] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.579103 4136 consensus_queue.cc:260] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.579187 3365 catalog_manager.cc:4841] TS c815aefee1744205a5197698c6db1272 (127.24.130.131:36025): tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 (table test-table [id=6145576b790948d79eb96b85448943c8]) successfully deleted
I20260806 02:51:07.579370 4136 ts_tablet_manager.cc:1428] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:07.579393 3365 catalog_manager.cc:4841] TS 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081): tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 (table test-table [id=6145576b790948d79eb96b85448943c8]) successfully deleted
I20260806 02:51:07.580932 3362 catalog_manager.cc:4841] TS eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915): tablet 0e95a7f6ec254a28ad53d2cf8b515ca1 (table test-table [id=6145576b790948d79eb96b85448943c8]) successfully deleted
I20260806 02:51:07.582502 4137 tablet_bootstrap.cc:492] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Bootstrap starting.
I20260806 02:51:07.583546 4137 tablet_bootstrap.cc:654] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.584805 4137 tablet_bootstrap.cc:492] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: No bootstrap required, opened a new log
I20260806 02:51:07.584929 4137 ts_tablet_manager.cc:1397] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:51:07.585605 4137 raft_consensus.cc:348] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.585767 4137 raft_consensus.cc:374] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.585814 4137 raft_consensus.cc:729] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee208549fa54101badd9a03a4e6a88c, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.585947 4137 consensus_queue.cc:260] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.586251 4137 ts_tablet_manager.cc:1428] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:07.614507 4147 raft_consensus.cc:482] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:07.614730 4147 raft_consensus.cc:504] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.615831 4147 leader_election.cc:288] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081), c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:07.620608 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.616676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:54699 (local address 127.24.130.131:36025)
0806 02:51:07.616808 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.616812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.617017 (+ 205us) server_negotiation.cc:407] Connection header received
0806 02:51:07.617118 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.617121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.617166 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.617245 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.617579 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.618308 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.619475 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.619974 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.620096 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.620155 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.620395 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.620477 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:51:07.620620 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.616538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:36025 (local address 127.24.130.129:54699)
0806 02:51:07.616908 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.616923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.617030 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.617285 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.617288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.617307 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.617445 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.617454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.618442 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.618447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.619316 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.619326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.620119 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.620122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.620242 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.620269 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.620356 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.620448 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":89,"threads_started":1}
I20260806 02:51:07.621176 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.616243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:37339 (local address 127.24.130.130:43081)
0806 02:51:07.616395 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.616400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.616564 (+ 164us) server_negotiation.cc:407] Connection header received
0806 02:51:07.616807 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.616811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.616867 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.616926 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.617386 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.618178 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.619888 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.620437 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.620555 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.620600 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.620958 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.621050 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:51:07.621356 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5e89cf0a264fb799289916373f6769" candidate_uuid: "eee208549fa54101badd9a03a4e6a88c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c815aefee1744205a5197698c6db1272" is_pre_election: true
I20260806 02:51:07.621610 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.616148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:43081 (local address 127.24.130.129:37339)
0806 02:51:07.616467 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.616485 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.616659 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.617101 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.617104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.617125 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.617275 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.617283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.618315 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.618318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.619732 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.619744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.620667 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.620672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.620773 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.620802 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.621355 (+ 553us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.621438 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260806 02:51:07.621642 4019 raft_consensus.cc:2454] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee208549fa54101badd9a03a4e6a88c in term 0.
I20260806 02:51:07.622089 3509 leader_election.cc:302] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c815aefee1744205a5197698c6db1272, eee208549fa54101badd9a03a4e6a88c; no voters:
I20260806 02:51:07.622123 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5e89cf0a264fb799289916373f6769" candidate_uuid: "eee208549fa54101badd9a03a4e6a88c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1117ae42c447474c991fdb7368235fdc" is_pre_election: true
I20260806 02:51:07.622274 3810 raft_consensus.cc:2454] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee208549fa54101badd9a03a4e6a88c in term 0.
I20260806 02:51:07.622390 4147 raft_consensus.cc:2781] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:07.622462 4147 raft_consensus.cc:482] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:07.622509 4147 raft_consensus.cc:3037] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.623662 4147 raft_consensus.cc:504] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.624107 4147 leader_election.cc:288] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [CANDIDATE]: Term 1 election: Requested vote from peers 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081), c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:07.624518 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5e89cf0a264fb799289916373f6769" candidate_uuid: "eee208549fa54101badd9a03a4e6a88c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1117ae42c447474c991fdb7368235fdc"
I20260806 02:51:07.624651 3810 raft_consensus.cc:3037] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.625846 3810 raft_consensus.cc:2454] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee208549fa54101badd9a03a4e6a88c in term 1.
I20260806 02:51:07.626147 3509 leader_election.cc:302] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, eee208549fa54101badd9a03a4e6a88c; no voters:
I20260806 02:51:07.626127 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5e89cf0a264fb799289916373f6769" candidate_uuid: "eee208549fa54101badd9a03a4e6a88c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c815aefee1744205a5197698c6db1272"
I20260806 02:51:07.626284 4019 raft_consensus.cc:3037] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.626394 4147 raft_consensus.cc:2781] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:07.626713 4147 raft_consensus.cc:686] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 1 LEADER]: Becoming Leader. State: Replica: eee208549fa54101badd9a03a4e6a88c, State: Running, Role: LEADER
I20260806 02:51:07.627013 4147 consensus_queue.cc:237] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.627581 4019 raft_consensus.cc:2454] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee208549fa54101badd9a03a4e6a88c in term 1.
I20260806 02:51:07.629568 3379 catalog_manager.cc:5495] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c reported cstate change: term changed from 0 to 1, leader changed from <none> to eee208549fa54101badd9a03a4e6a88c (127.24.130.129). New cstate: current_term: 1 leader_uuid: "eee208549fa54101badd9a03a4e6a88c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.190@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:07.679554 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.668702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:33627 (local address 127.0.0.1:54242)
0806 02:51:07.669133 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.669155 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.669319 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.669637 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.669640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.670310 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:07.670472 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.670481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.671338 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.671341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.672243 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.672257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.672761 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.672765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.672873 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.673671 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:07.673686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:07.675787 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:07.677707 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:07.677720 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:07.677726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:07.677982 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:07.678307 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:07.678312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:07.678314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:07.678404 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:07.678894 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:07.678903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:07.679029 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.679151 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.679338 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260806 02:51:07.679553 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.668782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.24.130.190:33627)
0806 02:51:07.668942 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.668947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.669268 (+ 321us) server_negotiation.cc:407] Connection header received
0806 02:51:07.669438 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.669441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.669500 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.669571 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.670630 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.671206 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.672394 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.672607 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.672717 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.675952 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:07.675956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:07.675958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:07.675998 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:07.677575 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.678102 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.678107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.678114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.678210 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.678505 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.678508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.678510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.678781 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:07.678894 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.679220 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.679396 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":68}
I20260806 02:51:07.684056 3379 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:54242:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:51:07.684837 3379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:07.690487 3988 tablet_service.cc:1467] Processing CreateTablet for tablet bac987dad8174903b6b7be423ce269f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cec082305ea8438ab715a18e86efb5ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:51:07.690428 3574 tablet_service.cc:1467] Processing CreateTablet for tablet bac987dad8174903b6b7be423ce269f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cec082305ea8438ab715a18e86efb5ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:51:07.690961 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac987dad8174903b6b7be423ce269f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.691104 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac987dad8174903b6b7be423ce269f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.693547 4139 tablet_bootstrap.cc:492] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: Bootstrap starting.
I20260806 02:51:07.694231 4137 tablet_bootstrap.cc:492] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: Bootstrap starting.
I20260806 02:51:07.694126 3785 tablet_service.cc:1467] Processing CreateTablet for tablet bac987dad8174903b6b7be423ce269f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cec082305ea8438ab715a18e86efb5ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:51:07.694536 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac987dad8174903b6b7be423ce269f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.694599 4139 tablet_bootstrap.cc:654] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.695245 4137 tablet_bootstrap.cc:654] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.696791 4137 tablet_bootstrap.cc:492] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: No bootstrap required, opened a new log
I20260806 02:51:07.696900 4137 ts_tablet_manager.cc:1397] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:07.697319 4136 tablet_bootstrap.cc:492] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: Bootstrap starting.
I20260806 02:51:07.697391 4137 raft_consensus.cc:348] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.697513 4137 raft_consensus.cc:374] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.697546 4137 raft_consensus.cc:729] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee208549fa54101badd9a03a4e6a88c, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.698210 4137 consensus_queue.cc:260] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.698438 4136 tablet_bootstrap.cc:654] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.698529 4147 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.698725 4147 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.699023 4137 ts_tablet_manager.cc:1428] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:51:07.699690 4139 tablet_bootstrap.cc:492] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: No bootstrap required, opened a new log
I20260806 02:51:07.699779 4139 ts_tablet_manager.cc:1397] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:51:07.700305 4139 raft_consensus.cc:348] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.700428 4139 raft_consensus.cc:374] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.700423 4136 tablet_bootstrap.cc:492] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: No bootstrap required, opened a new log
I20260806 02:51:07.700462 4139 raft_consensus.cc:729] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c815aefee1744205a5197698c6db1272, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.700510 4136 ts_tablet_manager.cc:1397] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 02:51:07.700587 4139 consensus_queue.cc:260] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.700863 4139 ts_tablet_manager.cc:1428] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:07.700942 4136 raft_consensus.cc:348] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.701056 4136 raft_consensus.cc:374] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.701092 4136 raft_consensus.cc:729] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.701102 4166 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.701227 4136 consensus_queue.cc:260] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.701272 4166 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.701502 4136 ts_tablet_manager.cc:1428] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:51:07.702025 4146 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.702179 4146 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.773432 4146 raft_consensus.cc:482] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:07.773609 4146 raft_consensus.cc:504] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.775004 4146 leader_election.cc:288] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:07.779251 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.775486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:32927 (local address 127.24.130.131:36025)
0806 02:51:07.775619 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.775623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.775792 (+ 169us) server_negotiation.cc:407] Connection header received
0806 02:51:07.776026 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.776028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.776070 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.776155 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.776480 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.776997 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.778188 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.778578 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.778733 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.778795 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.779008 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.779101 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:51:07.779256 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.775361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:57679 (local address 127.24.130.129:33915)
0806 02:51:07.775528 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.775532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.775804 (+ 272us) server_negotiation.cc:407] Connection header received
0806 02:51:07.776038 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.776041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.776091 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.776185 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:07.776502 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.776997 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.778163 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.778582 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.778749 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.778804 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.779032 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.779128 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:51:07.779296 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.775232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33915 (local address 127.24.130.130:57679)
0806 02:51:07.775688 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.775702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.775912 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.776195 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.776198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.776215 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.776385 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.776391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.777114 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.777118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.778030 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.778040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.778743 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.778747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.778902 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.778920 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.779012 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.779119 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":67,"spinlock_wait_cycles":2176,"thread_start_us":173,"threads_started":1}
I20260806 02:51:07.779300 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.775339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:36025 (local address 127.24.130.130:32927)
0806 02:51:07.775698 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.775710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.775905 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.776169 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.776172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.776187 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:07.776357 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.776362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.777112 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.777116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.778051 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.778061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.778722 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.778726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.778873 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.778899 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.778992 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.779106 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":171,"spinlock_wait_cycles":108288,"thread_start_us":123,"threads_started":1}
I20260806 02:51:07.779839 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac987dad8174903b6b7be423ce269f6" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c" is_pre_election: true
I20260806 02:51:07.780030 3599 raft_consensus.cc:2454] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 0.
I20260806 02:51:07.780085 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac987dad8174903b6b7be423ce269f6" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c815aefee1744205a5197698c6db1272" is_pre_election: true
I20260806 02:51:07.780246 4019 raft_consensus.cc:2454] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 0.
I20260806 02:51:07.780558 3706 leader_election.cc:302] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, eee208549fa54101badd9a03a4e6a88c; no voters:
I20260806 02:51:07.781035 4146 raft_consensus.cc:2781] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:07.781091 4146 raft_consensus.cc:482] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:07.781118 4146 raft_consensus.cc:3037] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.782184 4146 raft_consensus.cc:504] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.782580 4146 leader_election.cc:288] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 election: Requested vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:07.782909 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac987dad8174903b6b7be423ce269f6" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c"
I20260806 02:51:07.782990 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac987dad8174903b6b7be423ce269f6" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c815aefee1744205a5197698c6db1272"
I20260806 02:51:07.783052 3599 raft_consensus.cc:3037] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.783130 4019 raft_consensus.cc:3037] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:07.784035 3599 raft_consensus.cc:2454] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 1.
I20260806 02:51:07.784044 4019 raft_consensus.cc:2454] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 1.
I20260806 02:51:07.784372 3706 leader_election.cc:302] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, c815aefee1744205a5197698c6db1272, eee208549fa54101badd9a03a4e6a88c; no voters:
I20260806 02:51:07.784619 4146 raft_consensus.cc:2781] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:07.784884 4146 raft_consensus.cc:686] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [term 1 LEADER]: Becoming Leader. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Running, Role: LEADER
I20260806 02:51:07.785156 4146 consensus_queue.cc:237] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.786379 4205 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: TxnStatusTablet state changed. Reason: New leader 1117ae42c447474c991fdb7368235fdc. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.786514 4205 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:51:07.786819 4207 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:51:07.786978 4207 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:51:07.787711 3374 catalog_manager.cc:5495] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1117ae42c447474c991fdb7368235fdc (127.24.130.130). New cstate: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } health_report { overall_health: UNKNOWN } } }
I20260806 02:51:07.809839 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.805964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:43081 (local address 127.0.0.1:38826)
0806 02:51:07.806130 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.806151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.806272 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.806708 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.806711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.806719 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:51:07.806829 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.806834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.807563 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.807567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.808178 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.808186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.808673 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.808676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.808769 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.808818 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.809515 (+ 697us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.809534 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.809602 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.809693 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:51:07.809865 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.806097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.24.130.130:43081)
0806 02:51:07.806254 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.806259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.806275 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:51:07.806479 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.806481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.806536 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.806639 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:07.806944 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.807448 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.808325 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.808545 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.808656 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.808979 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.809412 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.809542 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.809626 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.809678 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:51:07.813163 3599 raft_consensus.cc:1261] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Refusing update from remote peer 1117ae42c447474c991fdb7368235fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:07.813552 4019 raft_consensus.cc:1261] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Refusing update from remote peer 1117ae42c447474c991fdb7368235fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:07.813738 4146 consensus_queue.cc:1035] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:07.814131 4205 consensus_queue.cc:1035] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:07.815979 4166 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: TxnStatusTablet state changed. Reason: New leader 1117ae42c447474c991fdb7368235fdc. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.816107 4166 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.816108 4147 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: TxnStatusTablet state changed. Reason: New leader 1117ae42c447474c991fdb7368235fdc. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.816226 4147 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.818193 4158 mvcc.cc:205] Tried to move back new op lower bound from 7315393199356338176 to 7315393199250337792. Current Snapshot: MvccSnapshot[applied={T|T < 7315393199356338176}]
I20260806 02:51:07.818176 4205 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.818311 4205 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:51:07.818809 4146 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.819044 4146 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P 1117ae42c447474c991fdb7368235fdc: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:51:07.819830 4166 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.819947 4166 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P c815aefee1744205a5197698c6db1272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.822285 4147 tablet_replica.cc:439] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } } }
I20260806 02:51:07.822425 4147 tablet_replica.cc:442] T bac987dad8174903b6b7be423ce269f6 P eee208549fa54101badd9a03a4e6a88c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:51:07.829479 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.825203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.24.130.129:33915)
0806 02:51:07.825333 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.825338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.825355 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:07.825528 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.825532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.825599 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.825721 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:07.825942 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.826764 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.827653 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.828010 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.828116 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.828487 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.829043 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.829179 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.829267 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.829327 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:51:07.841768 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.837683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:53711 (local address 127.24.130.130:43081)
0806 02:51:07.837864 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.837869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.838134 (+ 265us) server_negotiation.cc:407] Connection header received
0806 02:51:07.838237 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.838240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.838298 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.838396 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:07.838623 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.839486 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.840602 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.840845 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.840969 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.841305 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.841386 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.841471 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.841562 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.841617 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:51:07.841795 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.837591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:43081 (local address 127.24.130.129:53711)
0806 02:51:07.838045 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.838060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.838156 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.838403 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.838405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.838410 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:51:07.838517 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.838522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.839615 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.839618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.840429 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.840440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.840973 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.840977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.841081 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.841144 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.841475 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.841485 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.841552 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.841646 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":118,"threads_started":1}
I20260806 02:51:07.849473 4019 raft_consensus.cc:1261] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Refusing update from remote peer eee208549fa54101badd9a03a4e6a88c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:07.850101 3810 raft_consensus.cc:1261] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Refusing update from remote peer eee208549fa54101badd9a03a4e6a88c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:07.850380 4147 consensus_queue.cc:1035] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:07.850579 4182 consensus_queue.cc:1035] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:07.853713 4161 mvcc.cc:205] Tried to move back new op lower bound from 7315393199506927616 to 7315393198603214848. Current Snapshot: MvccSnapshot[applied={T|T < 7315393199506927616}]
I20260806 02:51:07.874202 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.870150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:54431 (local address 127.24.130.129:33915)
0806 02:51:07.870306 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.870313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.870507 (+ 194us) server_negotiation.cc:407] Connection header received
0806 02:51:07.870644 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.870689 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.870748 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.870816 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:07.871108 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.871773 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.873054 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.873312 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.873393 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.873728 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.873838 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.873891 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.874000 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.874055 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:51:07.874243 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.870060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33915 (local address 127.24.130.130:54431)
0806 02:51:07.870426 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:07.870444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:07.870562 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:07.870851 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:07.870854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:07.870862 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:51:07.871003 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.871009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.871897 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.871902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.872936 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:07.872948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.873428 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.873432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:07.873531 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.873592 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:51:07.873903 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:51:07.873910 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:51:07.873979 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:51:07.874062 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":147,"threads_started":1}
I20260806 02:51:07.900578 3378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:07.900786 3378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:07.903424 3378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Sending DeleteTablet for 3 replicas of tablet ec5e89cf0a264fb799289916373f6769
I20260806 02:51:07.904103 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5e89cf0a264fb799289916373f6769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:07 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260806 02:51:07.904302 4176 tablet_replica.cc:330] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: stopping tablet replica
I20260806 02:51:07.904299 3988 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5e89cf0a264fb799289916373f6769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:07 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:35052
I20260806 02:51:07.904418 4176 raft_consensus.cc:2229] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:51:07.904498 4173 tablet_replica.cc:330] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: stopping tablet replica
I20260806 02:51:07.904618 4173 raft_consensus.cc:2229] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:07.904738 4173 raft_consensus.cc:2258] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:07.904727 4176 raft_consensus.cc:2258] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:07.905228 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5e89cf0a264fb799289916373f6769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:07 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:38822
I20260806 02:51:07.905431 4174 tablet_replica.cc:330] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: stopping tablet replica
I20260806 02:51:07.905535 4174 raft_consensus.cc:2229] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:07.905647 4174 raft_consensus.cc:2258] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:07.906085 4176 ts_tablet_manager.cc:1905] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:07.906448 4174 ts_tablet_manager.cc:1905] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:07.906875 4173 ts_tablet_manager.cc:1905] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:07.909188 4174 ts_tablet_manager.cc:1918] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:51:07.909188 4176 ts_tablet_manager.cc:1918] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:51:07.909261 4176 log.cc:1196] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/ec5e89cf0a264fb799289916373f6769
I20260806 02:51:07.909271 4174 log.cc:1196] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/ec5e89cf0a264fb799289916373f6769
I20260806 02:51:07.909549 4176 ts_tablet_manager.cc:1939] T ec5e89cf0a264fb799289916373f6769 P eee208549fa54101badd9a03a4e6a88c: Deleting consensus metadata
I20260806 02:51:07.909565 4174 ts_tablet_manager.cc:1939] T ec5e89cf0a264fb799289916373f6769 P 1117ae42c447474c991fdb7368235fdc: Deleting consensus metadata
I20260806 02:51:07.909744 4173 ts_tablet_manager.cc:1918] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:51:07.909816 4173 log.cc:1196] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/ec5e89cf0a264fb799289916373f6769
I20260806 02:51:07.910318 3365 catalog_manager.cc:4841] TS 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081): tablet ec5e89cf0a264fb799289916373f6769 (table test-table [id=ddb77c6aa3bc4a6fafd966ee862d4121]) successfully deleted
I20260806 02:51:07.910424 3362 catalog_manager.cc:4841] TS eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915): tablet ec5e89cf0a264fb799289916373f6769 (table test-table [id=ddb77c6aa3bc4a6fafd966ee862d4121]) successfully deleted
I20260806 02:51:07.910076 4173 ts_tablet_manager.cc:1939] T ec5e89cf0a264fb799289916373f6769 P c815aefee1744205a5197698c6db1272: Deleting consensus metadata
I20260806 02:51:07.911260 3365 catalog_manager.cc:4841] TS c815aefee1744205a5197698c6db1272 (127.24.130.131:36025): tablet ec5e89cf0a264fb799289916373f6769 (table test-table [id=ddb77c6aa3bc4a6fafd966ee862d4121]) successfully deleted
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:51:07.945461 3448 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:07.936575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.24.130.190:33627)
0806 02:51:07.936721 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.936725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.937049 (+ 324us) server_negotiation.cc:407] Connection header received
0806 02:51:07.937171 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.937175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.937243 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.937352 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.938328 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.939297 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.940780 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.941127 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.941255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.945315 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54252: BlockingRecv error: recv got EOF from 127.0.0.1:54252 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:51:07 dist-test-slave-3kk6 krb5kdc[3320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM
I20260806 02:51:07.962226 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:07.951161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.24.130.190:33627)
0806 02:51:07.951310 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:07.951315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:07.951578 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:51:07.951703 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:07.951706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:07.951772 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:07.951869 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:07.952667 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.953467 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.954622 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:07.954994 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:07.955098 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:07.958257 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:07.958261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:07.958264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:07.958308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:07.960174 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.960785 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.960789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.960791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.960893 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:07.961217 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:07.961220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:07.961222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:07.961475 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:07.961613 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:07.961908 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 02:51:07.962048 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:51:07.972980 3378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:07.973505 3378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:07.980123 3785 tablet_service.cc:1467] Processing CreateTablet for tablet b38dfca223074699b9cacee7c1e77501 (DEFAULT_TABLE table=test-table [id=d3171bc1835e462e86b3b30c1a5e865d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.980473 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38dfca223074699b9cacee7c1e77501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.983530 3988 tablet_service.cc:1467] Processing CreateTablet for tablet b38dfca223074699b9cacee7c1e77501 (DEFAULT_TABLE table=test-table [id=d3171bc1835e462e86b3b30c1a5e865d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.983846 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38dfca223074699b9cacee7c1e77501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.983816 3574 tablet_service.cc:1467] Processing CreateTablet for tablet b38dfca223074699b9cacee7c1e77501 (DEFAULT_TABLE table=test-table [id=d3171bc1835e462e86b3b30c1a5e865d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:07.984131 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b38dfca223074699b9cacee7c1e77501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:07.986320 4136 tablet_bootstrap.cc:492] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Bootstrap starting.
I20260806 02:51:07.987206 4136 tablet_bootstrap.cc:654] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.987753 4139 tablet_bootstrap.cc:492] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Bootstrap starting.
I20260806 02:51:07.988586 4139 tablet_bootstrap.cc:654] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.989115 4137 tablet_bootstrap.cc:492] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Bootstrap starting.
I20260806 02:51:07.990020 4137 tablet_bootstrap.cc:654] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:07.991935 4139 tablet_bootstrap.cc:492] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: No bootstrap required, opened a new log
I20260806 02:51:07.991968 4136 tablet_bootstrap.cc:492] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: No bootstrap required, opened a new log
I20260806 02:51:07.992030 4139 ts_tablet_manager.cc:1397] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 02:51:07.992087 4136 ts_tablet_manager.cc:1397] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 02:51:07.992475 4139 raft_consensus.cc:348] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.992578 4137 tablet_bootstrap.cc:492] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: No bootstrap required, opened a new log
I20260806 02:51:07.992594 4139 raft_consensus.cc:374] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.992626 4139 raft_consensus.cc:729] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c815aefee1744205a5197698c6db1272, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.992556 4136 raft_consensus.cc:348] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.992656 4137 ts_tablet_manager.cc:1397] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:51:07.992674 4136 raft_consensus.cc:374] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.992707 4136 raft_consensus.cc:729] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.992754 4139 consensus_queue.cc:260] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.992837 4136 consensus_queue.cc:260] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.993017 4139 ts_tablet_manager.cc:1428] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:07.993093 4136 ts_tablet_manager.cc:1428] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:51:07.993144 4137 raft_consensus.cc:348] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.993263 4137 raft_consensus.cc:374] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:07.993314 4137 raft_consensus.cc:729] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee208549fa54101badd9a03a4e6a88c, State: Initialized, Role: FOLLOWER
I20260806 02:51:07.993427 4137 consensus_queue.cc:260] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:07.993676 4137 ts_tablet_manager.cc:1428] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:51:08.099045 4166 raft_consensus.cc:482] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:08.099228 4166 raft_consensus.cc:504] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.099795 4166 leader_election.cc:288] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081)
I20260806 02:51:08.100365 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38dfca223074699b9cacee7c1e77501" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c" is_pre_election: true
I20260806 02:51:08.100401 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38dfca223074699b9cacee7c1e77501" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1117ae42c447474c991fdb7368235fdc" is_pre_election: true
I20260806 02:51:08.100548 3599 raft_consensus.cc:2454] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 0.
I20260806 02:51:08.100560 3810 raft_consensus.cc:2454] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 0.
I20260806 02:51:08.100885 3921 leader_election.cc:302] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c815aefee1744205a5197698c6db1272, eee208549fa54101badd9a03a4e6a88c; no voters:
I20260806 02:51:08.101069 4166 raft_consensus.cc:2781] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:08.101147 4166 raft_consensus.cc:482] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:08.101202 4166 raft_consensus.cc:3037] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.102828 4166 raft_consensus.cc:504] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.103253 4166 leader_election.cc:288] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 election: Requested vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081)
I20260806 02:51:08.103664 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38dfca223074699b9cacee7c1e77501" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1117ae42c447474c991fdb7368235fdc"
I20260806 02:51:08.103816 3810 raft_consensus.cc:3037] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.104435 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b38dfca223074699b9cacee7c1e77501" candidate_uuid: "c815aefee1744205a5197698c6db1272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c"
I20260806 02:51:08.104553 3599 raft_consensus.cc:3037] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.104866 3810 raft_consensus.cc:2454] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 1.
I20260806 02:51:08.105172 3924 leader_election.cc:302] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, c815aefee1744205a5197698c6db1272; no voters:
I20260806 02:51:08.105437 4166 raft_consensus.cc:2781] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:08.105531 3599 raft_consensus.cc:2454] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c815aefee1744205a5197698c6db1272 in term 1.
I20260806 02:51:08.105657 4166 raft_consensus.cc:686] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 1 LEADER]: Becoming Leader. State: Replica: c815aefee1744205a5197698c6db1272, State: Running, Role: LEADER
I20260806 02:51:08.105837 4166 consensus_queue.cc:237] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.107573 3374 catalog_manager.cc:5495] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 reported cstate change: term changed from 0 to 1, leader changed from <none> to c815aefee1744205a5197698c6db1272 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "c815aefee1744205a5197698c6db1272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } health_report { overall_health: HEALTHY } } }
I20260806 02:51:08.124717 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.121394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.24.130.131:36025)
0806 02:51:08.121543 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:08.121547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:08.121563 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:51:08.121648 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:08.121651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:08.121705 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:08.121789 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:08.122094 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.122735 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.123602 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.123836 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.123955 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:08.124220 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:08.124303 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:08.124392 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:08.124513 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:51:08.124571 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:51:08.126825 3810 raft_consensus.cc:1261] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Refusing update from remote peer c815aefee1744205a5197698c6db1272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:08.126856 3599 raft_consensus.cc:1261] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Refusing update from remote peer c815aefee1744205a5197698c6db1272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:08.127332 4263 consensus_queue.cc:1035] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:08.127527 4166 consensus_queue.cc:1035] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:08.135828 3379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:08.136005 3379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:08.138078 3379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Sending DeleteTablet for 3 replicas of tablet b38dfca223074699b9cacee7c1e77501
I20260806 02:51:08.138641 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet b38dfca223074699b9cacee7c1e77501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:08 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260806 02:51:08.138820 3988 tablet_service.cc:1514] Processing DeleteTablet for tablet b38dfca223074699b9cacee7c1e77501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:08 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:35052
I20260806 02:51:08.138873 3785 tablet_service.cc:1514] Processing DeleteTablet for tablet b38dfca223074699b9cacee7c1e77501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:08 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:38822
I20260806 02:51:08.139045 4173 tablet_replica.cc:330] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: stopping tablet replica
I20260806 02:51:08.139057 4176 tablet_replica.cc:330] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: stopping tablet replica
I20260806 02:51:08.139199 4173 raft_consensus.cc:2229] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:51:08.139183 4176 raft_consensus.cc:2229] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:08.139364 4176 raft_consensus.cc:2258] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:08.139426 4173 raft_consensus.cc:2258] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:08.139560 4174 tablet_replica.cc:330] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: stopping tablet replica
I20260806 02:51:08.139523 3379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:51:08.139720 4174 raft_consensus.cc:2229] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:08.139858 4174 raft_consensus.cc:2258] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 02:51:08.139982 3379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:08.140208 4173 ts_tablet_manager.cc:1905] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:08.140642 4174 ts_tablet_manager.cc:1905] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:08.140772 4176 ts_tablet_manager.cc:1905] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:08.142573 4173 ts_tablet_manager.cc:1918] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:08.142632 4173 log.cc:1196] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/b38dfca223074699b9cacee7c1e77501
I20260806 02:51:08.142916 4173 ts_tablet_manager.cc:1939] T b38dfca223074699b9cacee7c1e77501 P c815aefee1744205a5197698c6db1272: Deleting consensus metadata
I20260806 02:51:08.143157 4174 ts_tablet_manager.cc:1918] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:08.143157 4176 ts_tablet_manager.cc:1918] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:08.143227 4174 log.cc:1196] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/b38dfca223074699b9cacee7c1e77501
I20260806 02:51:08.143227 4176 log.cc:1196] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/b38dfca223074699b9cacee7c1e77501
I20260806 02:51:08.143496 4174 ts_tablet_manager.cc:1939] T b38dfca223074699b9cacee7c1e77501 P 1117ae42c447474c991fdb7368235fdc: Deleting consensus metadata
I20260806 02:51:08.143519 4176 ts_tablet_manager.cc:1939] T b38dfca223074699b9cacee7c1e77501 P eee208549fa54101badd9a03a4e6a88c: Deleting consensus metadata
I20260806 02:51:08.143543 3365 catalog_manager.cc:4841] TS c815aefee1744205a5197698c6db1272 (127.24.130.131:36025): tablet b38dfca223074699b9cacee7c1e77501 (table test-table [id=d3171bc1835e462e86b3b30c1a5e865d]) successfully deleted
I20260806 02:51:08.144198 3365 catalog_manager.cc:4841] TS 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081): tablet b38dfca223074699b9cacee7c1e77501 (table test-table [id=d3171bc1835e462e86b3b30c1a5e865d]) successfully deleted
I20260806 02:51:08.144201 3362 catalog_manager.cc:4841] TS eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915): tablet b38dfca223074699b9cacee7c1e77501 (table test-table [id=d3171bc1835e462e86b3b30c1a5e865d]) successfully deleted
I20260806 02:51:08.147545 3785 tablet_service.cc:1467] Processing CreateTablet for tablet fd8d6e067e95466697a48f70d73f4a6e (DEFAULT_TABLE table=test-table [id=fb97982a5f85415a9237ab80a2b274ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:08.147547 3574 tablet_service.cc:1467] Processing CreateTablet for tablet fd8d6e067e95466697a48f70d73f4a6e (DEFAULT_TABLE table=test-table [id=fb97982a5f85415a9237ab80a2b274ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:08.147578 3988 tablet_service.cc:1467] Processing CreateTablet for tablet fd8d6e067e95466697a48f70d73f4a6e (DEFAULT_TABLE table=test-table [id=fb97982a5f85415a9237ab80a2b274ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:08.147832 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8d6e067e95466697a48f70d73f4a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:08.147832 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8d6e067e95466697a48f70d73f4a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:08.147836 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8d6e067e95466697a48f70d73f4a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:08.149664 4137 tablet_bootstrap.cc:492] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c: Bootstrap starting.
I20260806 02:51:08.150091 4139 tablet_bootstrap.cc:492] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Bootstrap starting.
I20260806 02:51:08.150122 4136 tablet_bootstrap.cc:492] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Bootstrap starting.
I20260806 02:51:08.150558 4137 tablet_bootstrap.cc:654] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:08.150846 4139 tablet_bootstrap.cc:654] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:08.150978 4136 tablet_bootstrap.cc:654] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:08.151772 4137 tablet_bootstrap.cc:492] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c: No bootstrap required, opened a new log
I20260806 02:51:08.151866 4137 ts_tablet_manager.cc:1397] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:51:08.152042 4139 tablet_bootstrap.cc:492] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: No bootstrap required, opened a new log
I20260806 02:51:08.152154 4139 ts_tablet_manager.cc:1397] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 02:51:08.152281 4137 raft_consensus.cc:348] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.152391 4137 raft_consensus.cc:374] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:08.152416 4137 raft_consensus.cc:729] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee208549fa54101badd9a03a4e6a88c, State: Initialized, Role: FOLLOWER
I20260806 02:51:08.152560 4137 consensus_queue.cc:260] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.152582 4139 raft_consensus.cc:348] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.152721 4139 raft_consensus.cc:374] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:08.152765 4139 raft_consensus.cc:729] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c815aefee1744205a5197698c6db1272, State: Initialized, Role: FOLLOWER
I20260806 02:51:08.152832 4137 ts_tablet_manager.cc:1428] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:51:08.152895 4139 consensus_queue.cc:260] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.153155 4139 ts_tablet_manager.cc:1428] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:51:08.154038 4136 tablet_bootstrap.cc:492] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: No bootstrap required, opened a new log
I20260806 02:51:08.154134 4136 ts_tablet_manager.cc:1397] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:51:08.154474 4136 raft_consensus.cc:348] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.154620 4136 raft_consensus.cc:374] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:08.154661 4136 raft_consensus.cc:729] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Initialized, Role: FOLLOWER
I20260806 02:51:08.155015 4136 consensus_queue.cc:260] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.155321 4136 ts_tablet_manager.cc:1428] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:51:08.333143 4205 raft_consensus.cc:482] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:08.333320 4205 raft_consensus.cc:504] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.333770 4205 leader_election.cc:288] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:08.334314 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8d6e067e95466697a48f70d73f4a6e" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c815aefee1744205a5197698c6db1272" is_pre_election: true
I20260806 02:51:08.334307 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8d6e067e95466697a48f70d73f4a6e" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c" is_pre_election: true
I20260806 02:51:08.334499 4019 raft_consensus.cc:2454] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 0.
I20260806 02:51:08.334519 3599 raft_consensus.cc:2454] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 0.
I20260806 02:51:08.334882 3709 leader_election.cc:302] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, c815aefee1744205a5197698c6db1272; no voters:
I20260806 02:51:08.335115 4205 raft_consensus.cc:2781] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:08.335199 4205 raft_consensus.cc:482] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:08.335258 4205 raft_consensus.cc:3037] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.336791 4205 raft_consensus.cc:504] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.337270 4205 leader_election.cc:288] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 election: Requested vote from peers eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915), c815aefee1744205a5197698c6db1272 (127.24.130.131:36025)
I20260806 02:51:08.337697 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8d6e067e95466697a48f70d73f4a6e" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c815aefee1744205a5197698c6db1272"
I20260806 02:51:08.337848 4019 raft_consensus.cc:3037] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.337836 3599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8d6e067e95466697a48f70d73f4a6e" candidate_uuid: "1117ae42c447474c991fdb7368235fdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee208549fa54101badd9a03a4e6a88c"
I20260806 02:51:08.337992 3599 raft_consensus.cc:3037] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.339277 4019 raft_consensus.cc:2454] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 1.
I20260806 02:51:08.339458 3599 raft_consensus.cc:2454] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1117ae42c447474c991fdb7368235fdc in term 1.
I20260806 02:51:08.339679 3709 leader_election.cc:302] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1117ae42c447474c991fdb7368235fdc, c815aefee1744205a5197698c6db1272; no voters:
I20260806 02:51:08.340005 4205 raft_consensus.cc:2781] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:08.340147 4205 raft_consensus.cc:686] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 1 LEADER]: Becoming Leader. State: Replica: 1117ae42c447474c991fdb7368235fdc, State: Running, Role: LEADER
I20260806 02:51:08.340303 4205 consensus_queue.cc:237] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } }
I20260806 02:51:08.342386 3379 catalog_manager.cc:5495] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1117ae42c447474c991fdb7368235fdc (127.24.130.130). New cstate: current_term: 1 leader_uuid: "1117ae42c447474c991fdb7368235fdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1117ae42c447474c991fdb7368235fdc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 43081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 } health_report { overall_health: UNKNOWN } } }
I20260806 02:51:08.400821 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.397819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.24.130.130:43081)
0806 02:51:08.398025 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:08.398029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:08.398042 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:51:08.398101 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:08.398103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:08.398144 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:08.398218 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:08.398431 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.399075 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.399879 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.400116 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.400209 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:08.400417 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:08.400504 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:08.400562 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:08.400645 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 02:51:08.400690 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260806 02:51:08.414316 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.407111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:43081 (local address 127.24.130.130:37843)
0806 02:51:08.411018 (+ 3907us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:08.411043 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:08.411148 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:08.411383 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:08.411385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:08.411391 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:51:08.411480 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:08.411485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.412515 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.412518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:08.413128 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:08.413136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.413685 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.413688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:08.413751 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:08.413777 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:51:08.414046 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:51:08.414050 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:51:08.414101 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 02:51:08.414168 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3802,"thread_start_us":124,"threads_started":1}
I20260806 02:51:08.414319 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.407193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:37843 (local address 127.24.130.130:43081)
0806 02:51:08.407309 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:08.407313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:08.411132 (+ 3819us) server_negotiation.cc:407] Connection header received
0806 02:51:08.411226 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:08.411229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:08.411284 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:08.411363 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:08.411575 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.412392 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.413237 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:08.413578 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:08.413673 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:08.413883 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:08.413964 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:08.414031 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:08.414109 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:51:08.414159 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":40}
I20260806 02:51:08.421030 3599 raft_consensus.cc:1261] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Refusing update from remote peer 1117ae42c447474c991fdb7368235fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:08.421073 4019 raft_consensus.cc:1261] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Refusing update from remote peer 1117ae42c447474c991fdb7368235fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:08.421497 4205 consensus_queue.cc:1035] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee208549fa54101badd9a03a4e6a88c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:08.421659 4146 consensus_queue.cc:1035] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c815aefee1744205a5197698c6db1272" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:08.457062 3374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:08.457260 3374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:08.459631 3374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f21ff689c40a4824bdd9d7c2c524c7b5: Sending DeleteTablet for 3 replicas of tablet fd8d6e067e95466697a48f70d73f4a6e
I20260806 02:51:08.460374 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8d6e067e95466697a48f70d73f4a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:08 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260806 02:51:08.460487 3988 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8d6e067e95466697a48f70d73f4a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:08 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:35052
I20260806 02:51:08.460634 3784 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8d6e067e95466697a48f70d73f4a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:08 UTC) from {username='oryx', principal='oryx/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:38822
I20260806 02:51:08.460708 4173 tablet_replica.cc:330] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: stopping tablet replica
I20260806 02:51:08.460712 4176 tablet_replica.cc:330] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c: stopping tablet replica
I20260806 02:51:08.460800 4174 tablet_replica.cc:330] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: stopping tablet replica
I20260806 02:51:08.460842 4173 raft_consensus.cc:2229] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:08.460841 4176 raft_consensus.cc:2229] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:08.460897 4174 raft_consensus.cc:2229] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:51:08.460970 4176 raft_consensus.cc:2258] T fd8d6e067e95466697a48f70d73f4a6e P eee208549fa54101badd9a03a4e6a88c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:08.460973 4173 raft_consensus.cc:2258] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:08.461241 4174 raft_consensus.cc:2258] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:08.461825 4173 ts_tablet_manager.cc:1905] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:08.461839 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 3488
I20260806 02:51:08.462172 4174 ts_tablet_manager.cc:1905] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:08.464617 4173 ts_tablet_manager.cc:1918] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:51:08.464617 4174 ts_tablet_manager.cc:1918] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:51:08.464694 4174 log.cc:1196] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/fd8d6e067e95466697a48f70d73f4a6e
I20260806 02:51:08.464702 4173 log.cc:1196] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/fd8d6e067e95466697a48f70d73f4a6e
I20260806 02:51:08.464972 4174 ts_tablet_manager.cc:1939] T fd8d6e067e95466697a48f70d73f4a6e P 1117ae42c447474c991fdb7368235fdc: Deleting consensus metadata
I20260806 02:51:08.464972 4173 ts_tablet_manager.cc:1939] T fd8d6e067e95466697a48f70d73f4a6e P c815aefee1744205a5197698c6db1272: Deleting consensus metadata
I20260806 02:51:08.470381 3365 catalog_manager.cc:4841] TS 1117ae42c447474c991fdb7368235fdc (127.24.130.130:43081): tablet fd8d6e067e95466697a48f70d73f4a6e (table test-table [id=fb97982a5f85415a9237ab80a2b274ee]) successfully deleted
I20260806 02:51:08.470593 3365 catalog_manager.cc:4841] TS c815aefee1744205a5197698c6db1272 (127.24.130.131:36025): tablet fd8d6e067e95466697a48f70d73f4a6e (table test-table [id=fb97982a5f85415a9237ab80a2b274ee]) successfully deleted
W20260806 02:51:08.480039 3362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:33915 (error 108)
I20260806 02:51:08.480275 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 3687
I20260806 02:51:08.482956 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.481113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:33915 (local address 127.0.0.1:47328)
0806 02:51:08.482603 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:08.482811 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:33915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1405,"thread_start_us":1280,"threads_started":1}
W20260806 02:51:08.487154 3362 catalog_manager.cc:4584] TS eee208549fa54101badd9a03a4e6a88c (127.24.130.129:33915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd8d6e067e95466697a48f70d73f4a6e: Network error: Client connection negotiation failed: client connection to 127.24.130.129:33915: connect: Connection refused (error 111)
I20260806 02:51:08.496771 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 3897
I20260806 02:51:08.507896 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 3334
2026-08-06T02:51:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3676 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:08 dist-test-slave-3kk6 krb5kdc[4294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:08 dist-test-slave-3kk6 krb5kdc[4294](info): set up 2 sockets
Aug 06 02:51:08 dist-test-slave-3kk6 krb5kdc[4294](info): commencing operation
krb5kdc: starting...
W20260806 02:51:09.745946 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:51:09 dist-test-slave-3kk6 krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:51:09.889693 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38227
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:35133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:10.012187 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:10.012559 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:10.012643 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:10.016814 4315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:10.016952 4315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:10.016990 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:10.017032 4315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:10.017093 4315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:10.022874 4315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:35133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:10.024178 4315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:10.025305 4315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:10.034592 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:10.034592 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:10.034850 4315 server_base.cc:1034] running on GCE node
W20260806 02:51:10.034633 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:10.035751 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:10.036964 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:10.038208 4315 hybrid_clock.cc:648] HybridClock initialized: now 1785984670038187 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:51:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1533 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:10 dist-test-slave-3kk6 krb5kdc[4333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:10 dist-test-slave-3kk6 krb5kdc[4333](info): set up 2 sockets
Aug 06 02:51:10 dist-test-slave-3kk6 krb5kdc[4333](info): commencing operation
krb5kdc: starting...
W20260806 02:51:11.267817 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:51:11 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:51:11.403383 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44967
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:11.521697 4352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:11.522033 4352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:11.522074 4352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:11.526386 4352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:11.526471 4352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:11.526492 4352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:11.526508 4352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:11.526525 4352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:11.532150 4352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:11.533361 4352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:11.534477 4352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:11.543520 4359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:11.543541 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:11.543541 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:11.544011 4352 server_base.cc:1034] running on GCE node
I20260806 02:51:11.544629 4352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:11.545737 4352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:11.546909 4352 hybrid_clock.cc:648] HybridClock initialized: now 1785984671546896 us; error 38 us; skew 500 ppm
Aug 06 02:51:11 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:11.553109 4352 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:51:11.554335 4352 webserver.cc:460] Webserver started at http://127.24.130.190:40375/ using document root <none> and password file <none>
I20260806 02:51:11.555055 4352 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:11.555124 4352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:11.555365 4352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:11.557669 4352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "70b7ba8d05f245e7a28fe2682a26a299"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "6a279fe59d7c73448e29536ebf2341c6"
server_key_iv: "e392ed7c309f464e1815a629146d261e"
server_key_version: "encryptionkey@0"
I20260806 02:51:11.558339 4352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "70b7ba8d05f245e7a28fe2682a26a299"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "6a279fe59d7c73448e29536ebf2341c6"
server_key_iv: "e392ed7c309f464e1815a629146d261e"
server_key_version: "encryptionkey@0"
I20260806 02:51:11.562578 4352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:51:11.565227 4371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:11.566543 4352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:51:11.566746 4352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "70b7ba8d05f245e7a28fe2682a26a299"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "6a279fe59d7c73448e29536ebf2341c6"
server_key_iv: "e392ed7c309f464e1815a629146d261e"
server_key_version: "encryptionkey@0"
I20260806 02:51:11.566891 4352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:11.582576 4352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:11.585814 4352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:11.586050 4352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:11.593819 4352 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:44967
I20260806 02:51:11.593847 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:44967 every 8 connection(s)
I20260806 02:51:11.594933 4352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:11.597960 4447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:11.601400 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 4352
I20260806 02:51:11.601527 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:51:11.601840 25098 external_mini_cluster.cc:1428] Setting key 400db5cfb756596ea403794495096bec
I20260806 02:51:11.604627 4447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Bootstrap starting.
I20260806 02:51:11.607690 4447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:11.608614 4447 log.cc:824] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:11.611161 4447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: No bootstrap required, opened a new log
Aug 06 02:51:11 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:11.614946 4447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:11.615196 4447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:11.615276 4447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b7ba8d05f245e7a28fe2682a26a299, State: Initialized, Role: FOLLOWER
I20260806 02:51:11.615779 4447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:11.615917 4447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:11.616008 4447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:11.616115 4447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:11.617122 4447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:11.617501 4447 leader_election.cc:302] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70b7ba8d05f245e7a28fe2682a26a299; no voters:
I20260806 02:51:11.617928 4447 leader_election.cc:288] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:11.618038 4453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:11.618271 4453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 LEADER]: Becoming Leader. State: Replica: 70b7ba8d05f245e7a28fe2682a26a299, State: Running, Role: LEADER
I20260806 02:51:11.618757 4453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:11.619149 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:11.603466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.24.130.190:44967)
0806 02:51:11.603946 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:11.603966 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:11.604011 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:51:11.604881 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:11.604908 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:11.605400 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:11.605800 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:11.606575 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:11.608138 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:11.609267 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:11.609802 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:11.610003 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:11.613042 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:11.613048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:11.613063 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:11.613092 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:11.615596 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:11.616159 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:11.616165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:11.616171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:11.616350 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:11.616643 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:11.616647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:11.616648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:11.617076 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:11.617297 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:11.617604 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:51:11.618001 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":323,"thread_start_us":129,"threads_started":1}
I20260806 02:51:11.619426 4447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:11.620010 4453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b7ba8d05f245e7a28fe2682a26a299. Latest consensus state: current_term: 1 leader_uuid: "70b7ba8d05f245e7a28fe2682a26a299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } } }
I20260806 02:51:11.620136 4453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:11.620808 4455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70b7ba8d05f245e7a28fe2682a26a299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } } }
I20260806 02:51:11.620991 4455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:11.621834 4462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:11.624866 4462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:11.631330 4462 catalog_manager.cc:1347] Generated new cluster ID: bfea8983bf9f413bbe701a9e093ded1c
I20260806 02:51:11.631413 4462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:11.642665 4462 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:11.643664 4462 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:11.650023 4462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Generated new TSK 0
I20260806 02:51:11.650914 4462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:51:11.683271 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44967
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:11.809247 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:11.809579 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:11.809626 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:11.814004 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:11.814110 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:11.814175 4487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:11.820045 4487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:11.821460 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:11.822638 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:11.833052 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:11.833040 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:11.833045 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:11.833413 4487 server_base.cc:1034] running on GCE node
I20260806 02:51:11.833935 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:11.834595 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:11.835845 4487 hybrid_clock.cc:648] HybridClock initialized: now 1785984671835830 us; error 29 us; skew 500 ppm
Aug 06 02:51:11 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:11.839650 4487 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:51:11.840634 4487 webserver.cc:460] Webserver started at http://127.24.130.129:35665/ using document root <none> and password file <none>
I20260806 02:51:11.841259 4487 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:11.841363 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:11.841625 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:11.843910 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "baca96583dd344ce84b76ff1fbc70f26"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "071da7de45f44d243cc9361d130a5793"
server_key_iv: "89b33d6038682b43518d47158ddd086e"
server_key_version: "encryptionkey@0"
I20260806 02:51:11.844547 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "baca96583dd344ce84b76ff1fbc70f26"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "071da7de45f44d243cc9361d130a5793"
server_key_iv: "89b33d6038682b43518d47158ddd086e"
server_key_version: "encryptionkey@0"
I20260806 02:51:11.849011 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:11.851643 4509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:11.852938 4487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:51:11.853104 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "baca96583dd344ce84b76ff1fbc70f26"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "071da7de45f44d243cc9361d130a5793"
server_key_iv: "89b33d6038682b43518d47158ddd086e"
server_key_version: "encryptionkey@0"
I20260806 02:51:11.853243 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:11.875468 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:11.879330 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:11.879582 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:11.880270 4487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:11.881188 4487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:11.881265 4487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:11.881340 4487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:11.881399 4487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:11.890998 4487 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:35841
I20260806 02:51:11.891023 4677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:35841 every 8 connection(s)
I20260806 02:51:11.892028 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
Aug 06 02:51:11 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984671, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:11.902370 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 4487
I20260806 02:51:11.902529 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:11.902906 25098 external_mini_cluster.cc:1428] Setting key 2d378df46fde670e16e31c3739207db9
I20260806 02:51:11.906760 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:11.893880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:38989 (local address 127.24.130.190:44967)
0806 02:51:11.894020 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:11.894024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:11.894890 (+ 866us) server_negotiation.cc:407] Connection header received
0806 02:51:11.896298 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:11.896302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:11.896367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:11.896474 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:11.897728 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:11.898279 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:11.899508 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:11.899746 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:11.899859 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:11.902642 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:11.902646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:11.902649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:11.902712 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:11.904290 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:11.904933 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:11.904936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:11.904938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:11.905041 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:11.905393 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:11.905399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:11.905403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:11.905668 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:11.905777 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:11.906311 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 02:51:11.906473 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:51:11.907761 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:11.894205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:38989)
0806 02:51:11.894720 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:11.894768 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:11.896069 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:11.896660 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:11.896667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:11.897058 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:11.897558 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:11.897574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:11.898433 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:11.898436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:11.899366 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:11.899384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:11.899923 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:11.899927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:11.900108 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:11.900825 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:11.900842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:11.902401 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:11.904423 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:11.904430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:11.904447 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:11.904806 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:11.905146 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:11.905149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:11.905150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:11.905298 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:11.905777 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:11.905782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:11.906057 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 02:51:11.906230 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:51:11.906521 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260806 02:51:11.909114 4678 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44967
I20260806 02:51:11.909389 4678 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:11.910050 4678 heartbeater.cc:511] Master 127.24.130.190:44967 requested a full tablet report, sending...
I20260806 02:51:11.912050 4397 ts_manager.cc:194] Registered new tserver with Master: baca96583dd344ce84b76ff1fbc70f26 (127.24.130.129:35841)
I20260806 02:51:11.913762 4397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:38989
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:51:11.935669 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44967
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:12.054420 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:12.054816 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:12.054924 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:12.059370 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:12.059485 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:12.059587 4684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:12.065146 4684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:12.066360 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:12.067479 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:12.077896 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.077899 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.077899 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:12.077992 4684 server_base.cc:1034] running on GCE node
I20260806 02:51:12.078557 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:12.079290 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:12.080504 4684 hybrid_clock.cc:648] HybridClock initialized: now 1785984672080490 us; error 40 us; skew 500 ppm
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:12.083792 4684 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:51:12.084760 4684 webserver.cc:460] Webserver started at http://127.24.130.130:42721/ using document root <none> and password file <none>
I20260806 02:51:12.085366 4684 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:12.085464 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:12.085709 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:12.087996 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "3fddefdd91724e5aa8a01ceb108e79d6"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "98557d1f6fd7915e67b42a85fbf48df4"
server_key_iv: "c4bca7968d561e3ba8c0fc35a2a810d1"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.088626 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "3fddefdd91724e5aa8a01ceb108e79d6"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "98557d1f6fd7915e67b42a85fbf48df4"
server_key_iv: "c4bca7968d561e3ba8c0fc35a2a810d1"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.092777 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:12.095620 4707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.097416 4684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:12.097570 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "3fddefdd91724e5aa8a01ceb108e79d6"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "98557d1f6fd7915e67b42a85fbf48df4"
server_key_iv: "c4bca7968d561e3ba8c0fc35a2a810d1"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.097735 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:12.113497 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:12.117239 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:12.117417 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:12.118007 4684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:12.119035 4684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:12.119105 4684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.119168 4684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:12.119202 4684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.129206 4684 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37921
I20260806 02:51:12.129226 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37921 every 8 connection(s)
I20260806 02:51:12.130209 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:12.134547 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 4684
I20260806 02:51:12.134716 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:12.135070 25098 external_mini_cluster.cc:1428] Setting key b27f573545fdbb744d9e00afd1dea7de
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:12.145375 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.132290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:34377 (local address 127.24.130.190:44967)
0806 02:51:12.132452 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:12.132456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:12.133344 (+ 888us) server_negotiation.cc:407] Connection header received
0806 02:51:12.135092 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:12.135095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:12.135176 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:12.135269 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:12.136554 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.137094 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.138283 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.138522 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.138639 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.141422 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:12.141428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:12.141434 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:12.141471 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:12.143180 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:12.143759 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:12.143763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:12.143764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:12.143845 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:12.144213 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:12.144216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:12.144218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:12.144417 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:12.144526 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:12.145059 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 02:51:12.145203 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:51:12.146409 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.132689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:34377)
0806 02:51:12.133179 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:12.133229 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:12.134827 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:12.135492 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:12.135502 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:12.135892 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:12.136387 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.136402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.137232 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.137235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.138143 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.138153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.138742 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.138749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.138915 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.139593 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:12.139610 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:12.141232 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:12.143312 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:12.143319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:12.143337 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:12.143634 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:12.143949 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:12.143953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:12.143955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:12.144108 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:12.144527 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:12.144535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:12.144807 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 02:51:12.145013 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:51:12.145367 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260806 02:51:12.147729 4869 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44967
I20260806 02:51:12.147989 4869 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
I20260806 02:51:12.148597 4869 heartbeater.cc:511] Master 127.24.130.190:44967 requested a full tablet report, sending...
I20260806 02:51:12.149734 4397 ts_manager.cc:194] Registered new tserver with Master: 3fddefdd91724e5aa8a01ceb108e79d6 (127.24.130.130:37921)
I20260806 02:51:12.150462 4397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:34377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:51:12.167526 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44967
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:12.290761 4878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:12.291070 4878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:12.291117 4878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:12.295377 4878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:12.295455 4878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:12.295519 4878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:12.301100 4878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:12.302273 4878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:12.303386 4878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:12.313673 4893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.313648 4889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.313648 4897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:12.314075 4878 server_base.cc:1034] running on GCE node
I20260806 02:51:12.314558 4878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:12.315361 4878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:12.316573 4878 hybrid_clock.cc:648] HybridClock initialized: now 1785984672316557 us; error 34 us; skew 500 ppm
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:12.319768 4878 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:51:12.320937 4878 webserver.cc:460] Webserver started at http://127.24.130.131:33593/ using document root <none> and password file <none>
I20260806 02:51:12.321615 4878 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:12.321727 4878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:12.321988 4878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:12.324277 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "1585313914584105974aa90b31689b39"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "d3a76cda331de44fb2db6584eb6b0981"
server_key_iv: "1de67e3d1ae3333809c6dc5328c6519f"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.324918 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "1585313914584105974aa90b31689b39"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "d3a76cda331de44fb2db6584eb6b0981"
server_key_iv: "1de67e3d1ae3333809c6dc5328c6519f"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.329039 4878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:51:12.331988 4906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.333330 4878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:12.333493 4878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "1585313914584105974aa90b31689b39"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "d3a76cda331de44fb2db6584eb6b0981"
server_key_iv: "1de67e3d1ae3333809c6dc5328c6519f"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.333629 4878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:12.353539 4878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:12.357265 4878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:12.357554 4878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:12.358258 4878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:12.359216 4878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:12.359293 4878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.359372 4878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:12.359416 4878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.369065 4878 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:44015
I20260806 02:51:12.369089 5060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:44015 every 8 connection(s)
I20260806 02:51:12.370090 4878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:12.376557 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 4878
I20260806 02:51:12.376721 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:12.377041 25098 external_mini_cluster.cc:1428] Setting key f98d46f01937ce6598f14faec14123ab
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:12.385624 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.372227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:33043 (local address 127.24.130.190:44967)
0806 02:51:12.372377 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:12.372381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:12.373335 (+ 954us) server_negotiation.cc:407] Connection header received
0806 02:51:12.374841 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:12.374844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:12.374917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:12.374999 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:12.376308 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.377145 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.378775 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.379009 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.379097 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.381888 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:12.381891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:12.381893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:12.381921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:12.383314 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:12.383922 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:12.383930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:12.383931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:12.384016 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:12.384404 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:12.384407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:12.384408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:12.384621 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:12.384738 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:12.385300 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 02:51:12.385448 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:51:12.386744 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.372680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:33043)
0806 02:51:12.373164 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:12.373211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:12.374522 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:12.375173 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:12.375180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:12.375618 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:12.376144 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.376159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.377281 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.377283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.378637 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.378651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.379172 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.379176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.379386 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.380020 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:12.380037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:12.381678 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:12.383469 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:12.383477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:12.383495 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:12.383789 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:12.384116 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:12.384120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:12.384122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:12.384285 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:12.384738 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:12.384746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:12.385030 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 02:51:12.385242 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 02:51:12.385631 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260806 02:51:12.388130 5063 heartbeater.cc:347] Connected to a master server at 127.24.130.190:44967
I20260806 02:51:12.388379 5063 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:12.388903 5063 heartbeater.cc:511] Master 127.24.130.190:44967 requested a full tablet report, sending...
I20260806 02:51:12.390141 4397 ts_manager.cc:194] Registered new tserver with Master: 1585313914584105974aa90b31689b39 (127.24.130.131:44015)
I20260806 02:51:12.390957 4397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:33043
I20260806 02:51:12.391472 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab.
Entry for principal oryx/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab.
WARNING: no policy specified for oryx/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab.
Entry for principal oryx/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab.
WARNING: no policy specified for oryx/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab.
Entry for principal oryx/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab.
I20260806 02:51:12.478245 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 4487
I20260806 02:51:12.494707 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 4684
I20260806 02:51:12.504745 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 4878
I20260806 02:51:12.521395 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 4352
I20260806 02:51:12.536485 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44967
--webserver_interface=127.24.130.190
--webserver_port=40375
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:12.655125 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:12.655433 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:12.655478 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:12.659768 5075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:12.659862 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:12.659883 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:12.659900 5075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:12.659917 5075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:12.665475 5075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=40375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:12.666712 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:12.667829 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:51:12.677726 5075 server_base.cc:1034] running on GCE node
W20260806 02:51:12.677603 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.677603 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.677618 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:12.678561 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:12.679942 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:12.681114 5075 hybrid_clock.cc:648] HybridClock initialized: now 1785984672681090 us; error 46 us; skew 500 ppm
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:12.684284 5075 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:51:12.685331 5075 webserver.cc:460] Webserver started at http://127.24.130.190:40375/ using document root <none> and password file <none>
I20260806 02:51:12.685928 5075 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:12.686038 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:12.690331 5075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:12.692821 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.694078 5075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:51:12.694219 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "70b7ba8d05f245e7a28fe2682a26a299"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "6a279fe59d7c73448e29536ebf2341c6"
server_key_iv: "e392ed7c309f464e1815a629146d261e"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.694892 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:12.714829 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:12.718588 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:12.718847 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:12.726622 5075 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:44967
I20260806 02:51:12.726639 5171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:44967 every 8 connection(s)
I20260806 02:51:12.727798 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:12.732311 5172 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:51:12.734189 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Bootstrap starting.
I20260806 02:51:12.734436 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 5075
I20260806 02:51:12.735016 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:35841
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:44967
--webserver_port=35665
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:12.752918 5172 log.cc:824] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:12.760335 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:51:12.760774 5172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Bootstrap complete.
I20260806 02:51:12.765326 5172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:12.765766 5172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b7ba8d05f245e7a28fe2682a26a299, State: Initialized, Role: FOLLOWER
I20260806 02:51:12.766351 5172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:12.766477 5172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:12.766566 5172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:12.766698 5172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:51:12.768877 5172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:12.769237 5172 leader_election.cc:302] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70b7ba8d05f245e7a28fe2682a26a299; no voters:
I20260806 02:51:12.769616 5172 leader_election.cc:288] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:51:12.769717 5177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:51:12.769927 5177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [term 2 LEADER]: Becoming Leader. State: Replica: 70b7ba8d05f245e7a28fe2682a26a299, State: Running, Role: LEADER
I20260806 02:51:12.770241 5177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } }
I20260806 02:51:12.770989 5172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:12.771842 5179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70b7ba8d05f245e7a28fe2682a26a299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } } }
I20260806 02:51:12.771996 5179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:12.772702 5182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b7ba8d05f245e7a28fe2682a26a299. Latest consensus state: current_term: 2 leader_uuid: "70b7ba8d05f245e7a28fe2682a26a299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b7ba8d05f245e7a28fe2682a26a299" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44967 } } }
I20260806 02:51:12.772819 5182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:12.773365 5189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:12.777092 5189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:12.777778 5189 catalog_manager.cc:1259] Loaded cluster ID: bfea8983bf9f413bbe701a9e093ded1c
I20260806 02:51:12.777845 5189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:12.780316 5189 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:12.781229 5189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70b7ba8d05f245e7a28fe2682a26a299: Loaded TSK: 0
I20260806 02:51:12.782554 5189 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:51:12.857594 5175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:12.857931 5175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:12.858011 5175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:12.862897 5175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:12.862993 5175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:12.863097 5175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:12.869364 5175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.129.keytab
--principal=oryx/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:35841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=35665
--tserver_master_addrs=127.24.130.190:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:12.870501 5175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:12.871695 5175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:12.882169 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.882174 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.882182 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:12.882910 5175 server_base.cc:1034] running on GCE node
I20260806 02:51:12.883385 5175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:12.884147 5175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:12.885353 5175 hybrid_clock.cc:648] HybridClock initialized: now 1785984672885324 us; error 57 us; skew 500 ppm
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:12.888543 5175 init.cc:377] Logged in from keytab as oryx/127.24.130.129@KRBTEST.COM (short username oryx)
I20260806 02:51:12.889564 5175 webserver.cc:460] Webserver started at http://127.24.130.129:35665/ using document root <none> and password file <none>
I20260806 02:51:12.890169 5175 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:12.890254 5175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:12.894761 5175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:12.897171 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.898711 5175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:51:12.898870 5175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "baca96583dd344ce84b76ff1fbc70f26"
format_stamp: "Formatted at 2026-08-06 02:51:11 on dist-test-slave-3kk6"
server_key: "071da7de45f44d243cc9361d130a5793"
server_key_iv: "89b33d6038682b43518d47158ddd086e"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.899564 5175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:12.915637 5175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:12.919203 5175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:12.919447 5175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:12.920171 5175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:12.921110 5175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:12.921187 5175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.921263 5175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:12.921317 5175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:12.931499 5175 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:35841
I20260806 02:51:12.931524 5389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:35841 every 8 connection(s)
I20260806 02:51:12.932616 5175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:12.933568 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 5175
I20260806 02:51:12.934115 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:37921
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:44967
--webserver_port=42721
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:12.949666 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.935470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:42809)
0806 02:51:12.936120 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:12.936189 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:12.938154 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:12.939503 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:12.939517 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:12.940127 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:12.940897 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.940923 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.942272 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.942277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.943939 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.943956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.944752 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.944759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.944999 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.945772 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:12.945799 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:12.948831 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":157,"threads_started":1}
W20260806 02:51:12.950001 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:12.935198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:42809 (local address 127.24.130.190:44967)
0806 02:51:12.935688 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:12.935705 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:12.936362 (+ 657us) server_negotiation.cc:407] Connection header received
0806 02:51:12.938616 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:12.938628 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:12.939037 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:12.939332 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:12.941121 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.942103 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.944083 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.944556 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.944687 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.949137 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:42809: BlockingRecv error: recv got EOF from 127.24.130.129:42809 (error 108)
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
W20260806 02:51:12.950570 5390 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:44967 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.130.190:44967: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:12.956660 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.951115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:57955)
0806 02:51:12.951249 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:12.951268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:12.951361 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:12.951627 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:12.951630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:12.951819 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:12.951924 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.951930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.952701 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.952706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.953265 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.953273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.953750 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.953753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.953856 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.954398 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:12.954410 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:12.956506 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:51:12.956784 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:12.951202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:57955 (local address 127.24.130.190:44967)
0806 02:51:12.951343 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:12.951356 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:12.951373 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:12.951462 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:12.951464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:12.951517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:12.951608 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:12.952046 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.952584 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.953394 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.953621 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.953734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.956670 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:57955: BlockingRecv error: recv got EOF from 127.24.130.129:57955 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:12 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:12.963063 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:12.957447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:48901)
0806 02:51:12.957683 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:12.957699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:12.957790 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:12.958072 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:12.958075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:12.958249 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:12.958324 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.958330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.959158 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.959163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.959700 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:12.959709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.960178 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.960181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:12.960257 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.960799 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:12.960821 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:12.962899 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173}
W20260806 02:51:12.963333 5390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:51:12.963464 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:12.957566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:48901 (local address 127.24.130.190:44967)
0806 02:51:12.957677 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:12.957681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:12.957745 (+ 64us) server_negotiation.cc:407] Connection header received
0806 02:51:12.957914 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:12.957917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:12.957966 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:12.958054 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:12.958451 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.959043 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.959829 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:12.960052 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:12.960162 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:12.963036 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:48901: BlockingRecv error: recv got EOF from 127.24.130.129:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260806 02:51:13.058403 5396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:13.058782 5396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:13.058867 5396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:13.063123 5396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:13.063226 5396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:13.063321 5396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:13.068953 5396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.130.keytab
--principal=oryx/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:37921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=42721
--tserver_master_addrs=127.24.130.190:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:13.070165 5396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:13.071375 5396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:13.081651 5405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:13.081661 5404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:13.081661 5407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:13.082120 5396 server_base.cc:1034] running on GCE node
I20260806 02:51:13.082561 5396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:13.083365 5396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:13.084592 5396 hybrid_clock.cc:648] HybridClock initialized: now 1785984673084573 us; error 37 us; skew 500 ppm
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984673, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:13.087743 5396 init.cc:377] Logged in from keytab as oryx/127.24.130.130@KRBTEST.COM (short username oryx)
I20260806 02:51:13.088681 5396 webserver.cc:460] Webserver started at http://127.24.130.130:42721/ using document root <none> and password file <none>
I20260806 02:51:13.089247 5396 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:13.089330 5396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:13.093523 5396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:13.095860 5414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.097162 5396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:13.097270 5396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "3fddefdd91724e5aa8a01ceb108e79d6"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "98557d1f6fd7915e67b42a85fbf48df4"
server_key_iv: "c4bca7968d561e3ba8c0fc35a2a810d1"
server_key_version: "encryptionkey@0"
I20260806 02:51:13.097826 5396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:13.114085 5396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:13.117257 5396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:13.117436 5396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:13.118031 5396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:13.118953 5396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:13.119005 5396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.119048 5396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:13.119068 5396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.128643 5396 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37921
I20260806 02:51:13.128677 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37921 every 8 connection(s)
I20260806 02:51:13.129724 5396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:13.133776 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 5396
I20260806 02:51:13.134256 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:44015
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:44967
--webserver_port=33593
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:51:13.143613 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.131820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:59029 (local address 127.24.130.190:44967)
0806 02:51:13.131987 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.131992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.132950 (+ 958us) server_negotiation.cc:407] Connection header received
0806 02:51:13.134558 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.134562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.134639 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.134816 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.136250 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.136808 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.138813 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.139042 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.139132 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.143477 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:59029: BlockingRecv error: recv got EOF from 127.24.130.130:59029 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:13.144299 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.132269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:59029)
0806 02:51:13.132779 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.132827 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.134274 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.135020 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.135028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.135436 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.136059 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.136085 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.136954 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.136960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.138636 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.138688 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.139236 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.139244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.139556 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.140311 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.140340 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.143317 (+ 2977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
W20260806 02:51:13.144942 5585 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:44967 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.130.190:44967: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:13.151110 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.145516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:50021)
0806 02:51:13.145643 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.145660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.145754 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.146021 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.146023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.146205 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.146301 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.146307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.147142 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.147145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.147723 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.147731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.148205 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.148208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.148302 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.148835 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.148856 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.150956 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:51:13.151331 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.145640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:50021 (local address 127.24.130.190:44967)
0806 02:51:13.145770 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.145775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.145795 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:51:13.145851 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.145854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.145911 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.146002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.146426 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.147026 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.147851 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.148081 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.148191 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.151087 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:50021: BlockingRecv error: recv got EOF from 127.24.130.130:50021 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:13.157533 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.151726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:46223)
0806 02:51:13.151823 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.151840 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.151923 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.152206 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.152208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.152335 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.152410 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.152414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.153521 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.153523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.154031 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.154037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.154720 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.154724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.154791 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.155402 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.155413 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.157363 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 02:51:13.157753 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.151783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:46223 (local address 127.24.130.190:44967)
0806 02:51:13.151907 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.151912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.151929 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:13.152033 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.152037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.152084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.152173 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.152517 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.153412 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.154142 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.154542 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.154670 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.157622 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:46223: BlockingRecv error: recv got EOF from 127.24.130.130:46223 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:51:13.157943 5585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:51:13.264182 5588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:13.264506 5588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:13.264554 5588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:13.268646 5588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:13.268730 5588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:13.268795 5588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:13.275007 5588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/krb5kdc/oryx_127.24.130.131.keytab
--principal=oryx/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:44015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=33593
--tserver_master_addrs=127.24.130.190:44967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:13.276129 5588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:13.277141 5588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:13.287379 5598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:13.287495 5588 server_base.cc:1034] running on GCE node
W20260806 02:51:13.287381 5595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:13.287381 5594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:13.288261 5588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:13.288980 5588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:13.290169 5588 hybrid_clock.cc:648] HybridClock initialized: now 1785984673290157 us; error 29 us; skew 500 ppm
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984673, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:13.293160 5588 init.cc:377] Logged in from keytab as oryx/127.24.130.131@KRBTEST.COM (short username oryx)
I20260806 02:51:13.294090 5588 webserver.cc:460] Webserver started at http://127.24.130.131:33593/ using document root <none> and password file <none>
I20260806 02:51:13.294734 5588 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:13.294859 5588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:13.299008 5588 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:13.301245 5607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.302651 5588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:13.302820 5588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "1585313914584105974aa90b31689b39"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-3kk6"
server_key: "d3a76cda331de44fb2db6584eb6b0981"
server_key_iv: "1de67e3d1ae3333809c6dc5328c6519f"
server_key_version: "encryptionkey@0"
I20260806 02:51:13.303426 5588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:13.324824 5588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:13.328060 5588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:13.328299 5588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:13.328955 5588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:13.329842 5588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:13.329923 5588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.330000 5588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:13.330044 5588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.339507 5588 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:44015
I20260806 02:51:13.339529 5775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:44015 every 8 connection(s)
I20260806 02:51:13.340600 5588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:13.343668 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 5588
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:51:13.354712 5392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.342490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:54723 (local address 127.24.130.190:44967)
0806 02:51:13.342637 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.342641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.343514 (+ 873us) server_negotiation.cc:407] Connection header received
0806 02:51:13.345251 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.345253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.345315 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.345410 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.346760 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.347461 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.349123 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.349479 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.349601 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.354522 (+ 4921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:54723: BlockingRecv error: recv got EOF from 127.24.130.131:54723 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:51:13.355748 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.342881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:54723)
0806 02:51:13.343347 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.343393 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.345018 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.345570 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.345577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.345942 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.346572 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.346591 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.347609 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.347612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.348968 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.348983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.349719 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.349728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.350087 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.350793 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.350820 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.354326 (+ 3506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260806 02:51:13.356629 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.344816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.24.130.190:44967)
0806 02:51:13.345100 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.345104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.345119 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:13.345168 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.345174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.345233 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.345328 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.346149 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.346926 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.347976 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.348342 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.348463 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.350346 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:13.350357 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:13.350382 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:13.350421 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:13.353425 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:13.354107 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:13.354114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:13.354121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:13.354324 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:13.355001 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:13.355004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:13.355006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:13.355502 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:13.355723 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:13.356204 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 02:51:13.356505 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20260806 02:51:13.359578 5776 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:44967 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.130.190:44967: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:13.366379 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.360189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:41913)
0806 02:51:13.360333 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.360350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.360449 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.360799 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.360803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.361074 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.361244 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.361251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.362243 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.362246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.363138 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.363151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.363733 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.363736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.363820 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.364316 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.364327 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.366258 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:51:13.366529 5780 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.360350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:41913 (local address 127.24.130.190:44967)
0806 02:51:13.360497 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.360504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.360523 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:51:13.360570 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.360576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.360638 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.360741 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.361400 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.362114 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.363297 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.363609 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.363695 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.366406 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:41913: BlockingRecv error: recv got EOF from 127.24.130.131:41913 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:13.372130 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.367029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:42109)
0806 02:51:13.367173 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.367189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.367279 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.367541 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.367543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.367680 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.367756 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.367760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.368427 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.368429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.368958 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.368966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.369348 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.369351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.369409 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.369878 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.369889 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.371994 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:51:13.372438 5776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:51:13.372669 5780 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.367204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:42109 (local address 127.24.130.190:44967)
0806 02:51:13.367355 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.367358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.367370 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:51:13.367405 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.367407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.367442 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.367508 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.367853 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.368336 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.369054 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.369254 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.369326 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.372190 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:42109: BlockingRecv error: recv got EOF from 127.24.130.131:42109 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:13 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:13.969671 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.964058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:47917)
0806 02:51:13.964409 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:13.964425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:13.964533 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:13.964801 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:13.964803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:13.965003 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:13.965107 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.965112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.965889 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.965891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.966480 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:13.966490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.966946 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.966949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:13.967014 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.967563 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:13.967575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:13.969538 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
W20260806 02:51:13.969843 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:13.964211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47917 (local address 127.24.130.190:44967)
0806 02:51:13.964476 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.964479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.964494 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:13.964630 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.964632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.964676 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.964756 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.965227 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.965775 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.966592 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.966840 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.966918 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.969690 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:47917: BlockingRecv error: recv got EOF from 127.24.130.129:47917 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
Aug 06 02:51:14 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:14 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:14.165495 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:14.158792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:54181)
0806 02:51:14.159075 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:14.159090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:14.159202 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:14.159465 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:14.159467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:14.159665 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:14.159834 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:14.159841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.160600 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.160619 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:14.161626 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:14.161638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.162080 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.162082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:14.162156 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:14.162822 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:14.162835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:14.165321 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260806 02:51:14.165599 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:14.158885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:54181 (local address 127.24.130.190:44967)
0806 02:51:14.159046 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:14.159049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:14.159168 (+ 119us) server_negotiation.cc:407] Connection header received
0806 02:51:14.159291 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:14.159293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:14.159339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:14.159426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:14.159948 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.160479 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.161748 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.161972 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.162059 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:14.165476 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:54181: BlockingRecv error: recv got EOF from 127.24.130.130:54181 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 02:51:14 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:14 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:14.379449 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:14.373097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:38769)
0806 02:51:14.373444 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:14.373477 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:14.373638 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:14.373973 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:14.373976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:14.374210 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:14.374307 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:14.374313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.375170 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.375175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:14.375760 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:14.375768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.376240 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.376244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:14.376333 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:14.376892 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:14.376904 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:14.379279 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":123,"threads_started":1}
W20260806 02:51:14.379769 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:14.373274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38769 (local address 127.24.130.190:44967)
0806 02:51:14.373444 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:14.373450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:14.373570 (+ 120us) server_negotiation.cc:407] Connection header received
0806 02:51:14.373790 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:14.373794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:14.373852 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:14.373947 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:14.374437 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.375055 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.375889 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.376120 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.376242 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:14.379477 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:38769: BlockingRecv error: recv got EOF from 127.24.130.131:38769 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 02:51:14 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:14 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:14.976683 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:14.970567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:44101)
0806 02:51:14.970923 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:14.970940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:14.971085 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:14.971330 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:14.971332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:14.971541 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:14.971643 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:14.971648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.972606 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.972609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:14.973171 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:14.973229 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.973676 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.973678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:14.973825 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:14.974431 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:14.974443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:14.976456 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
W20260806 02:51:14.976794 5788 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:14.970748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:44101 (local address 127.24.130.190:44967)
0806 02:51:14.971089 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:14.971092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:14.971107 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:14.971166 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:14.971169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:14.971212 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:14.971292 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:14.971819 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.972485 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.973345 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:14.973572 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:14.973656 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:14.976637 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:44101: BlockingRecv error: recv got EOF from 127.24.130.129:44101 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
Aug 06 02:51:15 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:15 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:15.173101 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:15.166518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:40361)
0806 02:51:15.166893 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:15.166912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:15.167085 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:15.167386 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:15.167389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:15.167639 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:15.167737 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:15.167743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.168539 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.168544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:15.169110 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:15.169118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.169592 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.169596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:15.169683 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:15.170370 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:15.170411 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:15.172938 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
W20260806 02:51:15.173413 5788 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:15.166710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:40361 (local address 127.24.130.190:44967)
0806 02:51:15.166898 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:15.166905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:15.167012 (+ 107us) server_negotiation.cc:407] Connection header received
0806 02:51:15.167213 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:15.167215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:15.167265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:15.167362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:15.167868 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.168420 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.169236 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.169471 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.169587 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:15.173134 (+ 3547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:40361: BlockingRecv error: recv got EOF from 127.24.130.130:40361 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 02:51:15 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:15 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:15.386461 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:15.380493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:60911)
0806 02:51:15.380803 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:15.380828 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:15.380932 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:15.381189 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:15.381191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:15.381393 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:15.381491 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:15.381496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.382247 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.382249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:15.382841 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:15.382849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.383356 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.383358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:15.383426 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:15.384129 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:15.384145 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:15.386314 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
W20260806 02:51:15.386626 5788 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:15.380703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:60911 (local address 127.24.130.190:44967)
0806 02:51:15.380913 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:15.380916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:15.380934 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:51:15.381026 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:15.381028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:15.381077 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:15.381154 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:15.381602 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.382150 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.382952 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.383216 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.383346 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:15.386465 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:60911: BlockingRecv error: recv got EOF from 127.24.130.131:60911 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 06 02:51:15 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:15 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:15.984028 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:15.977730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:42483)
0806 02:51:15.978051 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:15.978073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:15.978220 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:15.978482 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:15.978484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:15.978721 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:15.978824 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:15.978829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.979582 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.979584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:15.980138 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:15.980146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.980650 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.980654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:15.980736 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:15.981305 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:15.981318 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:15.983901 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
W20260806 02:51:15.984229 5795 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:15.977903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:42483 (local address 127.24.130.190:44967)
0806 02:51:15.978232 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:15.978236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:15.978278 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:51:15.978321 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:15.978324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:15.978366 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:15.978443 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:15.978937 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.979482 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.980287 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:15.980521 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:15.980610 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:15.984047 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:42483: BlockingRecv error: recv got EOF from 127.24.130.129:42483 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
Aug 06 02:51:16 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:16 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:16.180251 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:16.174099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:37429)
0806 02:51:16.174517 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:16.174534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:16.174722 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:16.175031 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:16.175035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:16.175287 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:16.175384 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:16.175390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.176255 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.176260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:16.176844 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:16.176852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.177322 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.177326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:16.177414 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:16.178045 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:16.178057 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:16.180063 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":118,"threads_started":1}
W20260806 02:51:16.180339 5795 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:16.174269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:37429 (local address 127.24.130.190:44967)
0806 02:51:16.174483 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:16.174487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:16.174639 (+ 152us) server_negotiation.cc:407] Connection header received
0806 02:51:16.174853 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:16.174856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:16.174909 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:16.175002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:16.175516 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.176137 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.176972 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.177204 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.177318 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:16.180220 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:37429: BlockingRecv error: recv got EOF from 127.24.130.130:37429 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 06 02:51:16 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:16 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:16.393512 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:16.387399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:47131)
0806 02:51:16.387733 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:16.387750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:16.387867 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:16.388167 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:16.388170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:16.388377 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:16.388482 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:16.388487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.389251 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.389254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:16.389861 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:16.389870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.390305 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.390308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:16.390376 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:16.391075 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:16.391097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:16.393393 (+ 2296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
W20260806 02:51:16.393704 5795 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:16.387563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:47131 (local address 127.24.130.190:44967)
0806 02:51:16.387761 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:16.387765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:16.387832 (+ 67us) server_negotiation.cc:407] Connection header received
0806 02:51:16.387956 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:16.387959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:16.388010 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:16.388087 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:16.388599 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.389123 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.389972 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.390187 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.390266 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:16.393539 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:47131: BlockingRecv error: recv got EOF from 127.24.130.131:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 02:51:16 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:16 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:16.995218 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:16.985115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:57875)
0806 02:51:16.985460 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:16.985482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:16.985616 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:16.985901 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:16.985904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:16.986182 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:16.986315 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:16.986324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.987212 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.987218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:16.987799 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:16.987808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.988293 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.988296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:16.988375 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:16.992907 (+ 4532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:16.992923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:16.995039 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":121,"threads_started":1}
W20260806 02:51:16.995404 5800 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:16.985337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:57875 (local address 127.24.130.190:44967)
0806 02:51:16.985634 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:16.985638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:16.985652 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:51:16.985713 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:16.985715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:16.985757 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:16.985873 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:16.986458 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.987096 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.987932 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:16.988173 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:16.988295 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:16.995246 (+ 6951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:57875: BlockingRecv error: recv got EOF from 127.24.130.129:57875 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
Aug 06 02:51:17 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:17 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:17.186899 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:17.181107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:55719)
0806 02:51:17.181423 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:17.181438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:17.181540 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:17.181783 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:17.181785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:17.181987 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:17.182086 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:17.182091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.182926 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.182928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:17.183477 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:17.183484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.183986 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.183988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:17.184051 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:17.184584 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:17.184596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:17.186766 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":138,"threads_started":1}
W20260806 02:51:17.187120 5800 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:17.181177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:55719 (local address 127.24.130.190:44967)
0806 02:51:17.181340 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:17.181344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:17.181516 (+ 172us) server_negotiation.cc:407] Connection header received
0806 02:51:17.181623 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:17.181625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:17.181672 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:17.181747 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:17.182186 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.182822 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.183607 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.183873 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.183969 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:17.186987 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:55719: BlockingRecv error: recv got EOF from 127.24.130.130:55719 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:51:17 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:17 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:17.400799 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:17.394557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:34891)
0806 02:51:17.394905 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:17.394921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:17.395074 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:17.395428 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:17.395430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:17.395711 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:17.395851 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:17.395859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.396854 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.396857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:17.397429 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:17.397438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.397890 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.397892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:17.397955 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:17.398577 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:17.398590 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:17.400678 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":141,"threads_started":1}
W20260806 02:51:17.400997 5800 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:17.394754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:34891 (local address 127.24.130.190:44967)
0806 02:51:17.394909 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:17.394914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:17.395040 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:51:17.395206 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:17.395209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:17.395266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:17.395387 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:17.396024 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.396736 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.397560 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.397782 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.397873 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:17.400857 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:34891: BlockingRecv error: recv got EOF from 127.24.130.131:34891 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 02:51:17 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:17 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:18.002789 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:17.996259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:41469)
0806 02:51:17.996616 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:17.996641 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:17.996735 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:17.997058 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:17.997060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:17.997275 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:17.997380 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:17.997385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.998285 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.998288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:17.999051 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:17.999064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.999568 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.999571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:17.999643 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:18.000312 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:18.000324 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:18.002591 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
W20260806 02:51:18.003063 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:17.996388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:41469 (local address 127.24.130.190:44967)
0806 02:51:17.996677 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:17.996682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:17.996704 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:51:17.996849 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:17.996851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:17.996896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:17.996978 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:17.997527 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.998138 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.999197 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.999441 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.999536 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:18.002906 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:41469: BlockingRecv error: recv got EOF from 127.24.130.129:41469 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
Aug 06 02:51:18 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:18 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:18.194581 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:18.188136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:38027)
0806 02:51:18.188497 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:18.188513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:18.188632 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:18.188899 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:18.188901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:18.189112 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:18.189244 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:18.189250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.189998 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.190000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:18.190773 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:18.190784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.191378 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.191381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:18.191504 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:18.192273 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:18.192287 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:18.194453 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
W20260806 02:51:18.194836 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:18.188364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:38027 (local address 127.24.130.190:44967)
0806 02:51:18.188535 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:18.188538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:18.188578 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:51:18.188725 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:18.188728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:18.188772 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:18.188852 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:18.189356 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.189885 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.190884 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.191232 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.191340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:18.194708 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:38027: BlockingRecv error: recv got EOF from 127.24.130.130:38027 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 02:51:18 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:18 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:18.408304 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:18.401955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:47341)
0806 02:51:18.402323 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:18.402339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:18.402494 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:18.402897 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:18.402900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:18.403222 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:18.403323 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:18.403330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.404191 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.404200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:18.404788 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:18.404798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.405304 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.405307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:18.405380 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:18.406043 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:18.406061 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:18.408176 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":102,"threads_started":1}
W20260806 02:51:18.408517 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:18.402035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:47341 (local address 127.24.130.190:44967)
0806 02:51:18.402195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:18.402200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:18.402462 (+ 262us) server_negotiation.cc:407] Connection header received
0806 02:51:18.402724 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:18.402727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:18.402780 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:18.402873 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:18.403452 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.404014 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.404924 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:18.405152 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:18.405250 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:18.408366 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:47341: BlockingRecv error: recv got EOF from 127.24.130.131:47341 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 02:51:19 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:19 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:19.013727 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:19.003700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:36903)
0806 02:51:19.003997 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:19.004021 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:19.004119 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:19.004391 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:19.004393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:19.004597 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:19.004696 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:19.004701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.005463 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.005465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:19.006024 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:19.006032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.006461 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.006464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:19.006525 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.011504 (+ 4979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:19.011519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:19.013601 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
W20260806 02:51:19.013981 5811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:19.003793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:36903 (local address 127.24.130.190:44967)
0806 02:51:19.004054 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:19.004057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:19.004074 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:19.004211 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:19.004213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:19.004260 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:19.004349 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:19.004807 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.005360 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.006132 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.006358 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.006437 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.013804 (+ 7367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:36903: BlockingRecv error: recv got EOF from 127.24.130.129:36903 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
Aug 06 02:51:19 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:19 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:19.201344 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:19.195478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:55963)
0806 02:51:19.195827 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:19.195842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:19.195953 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:19.196212 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:19.196214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:19.196433 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:19.196535 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:19.196541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.197315 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.197318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:19.197892 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:19.197900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.198346 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.198348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:19.198422 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.199046 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:19.199058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:19.201175 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
W20260806 02:51:19.201596 5811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:19.195602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:55963 (local address 127.24.130.190:44967)
0806 02:51:19.195755 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:19.195758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:19.195919 (+ 161us) server_negotiation.cc:407] Connection header received
0806 02:51:19.196039 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:19.196042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:19.196088 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:19.196175 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:19.196654 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.197203 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.198010 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.198235 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.198318 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.201453 (+ 3135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:55963: BlockingRecv error: recv got EOF from 127.24.130.130:55963 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:51:19 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:19 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:19.415812 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:19.409303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:49485)
0806 02:51:19.409768 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:19.409786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:19.409971 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:19.410295 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:19.410298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:19.410542 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:19.410641 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:19.410698 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.411520 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.411525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:19.412109 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:19.412118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.412653 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.412657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:19.412763 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.413457 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:19.413469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:19.415684 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":122,"threads_started":1}
W20260806 02:51:19.416043 5811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:19.409473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:49485 (local address 127.24.130.190:44967)
0806 02:51:19.409700 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:19.409705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:19.409900 (+ 195us) server_negotiation.cc:407] Connection header received
0806 02:51:19.410088 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:19.410091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:19.410158 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:19.410262 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:19.410829 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.411396 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.412246 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.412489 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.412631 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.415881 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:49485: BlockingRecv error: recv got EOF from 127.24.130.131:49485 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 02:51:20 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:20 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:20.020184 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:20.014575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:49345)
0806 02:51:20.015002 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:20.015018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:20.015123 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:20.015377 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:20.015379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:20.015578 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:20.015676 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:20.015681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.016455 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.016457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:20.017010 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:20.017017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.017431 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.017434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:20.017496 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:20.018122 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:20.018135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:20.020065 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":97,"threads_started":1}
W20260806 02:51:20.020409 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:20.014697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:49345 (local address 127.24.130.190:44967)
0806 02:51:20.014976 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:20.014979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:20.015096 (+ 117us) server_negotiation.cc:407] Connection header received
0806 02:51:20.015208 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:20.015210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:20.015256 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:20.015339 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:20.015780 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.016354 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.017115 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.017333 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.017411 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:20.020218 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:49345: BlockingRecv error: recv got EOF from 127.24.130.129:49345 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":68,"threads_started":1}
Aug 06 02:51:20 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:20 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:20.209126 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:20.202303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:40145)
0806 02:51:20.202822 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:20.202839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:20.202965 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:20.203240 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:20.203242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:20.203456 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:20.203568 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:20.203574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.204614 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.204617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:20.205340 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:20.205351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.205996 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.205999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:20.206064 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:20.206950 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:20.206963 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:20.209005 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":111,"threads_started":1}
W20260806 02:51:20.209285 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:20.202446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:40145 (local address 127.24.130.190:44967)
0806 02:51:20.202643 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:20.202647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:20.202928 (+ 281us) server_negotiation.cc:407] Connection header received
0806 02:51:20.203058 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:20.203061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:20.203113 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:20.203200 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:20.203830 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.204465 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.205490 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.205858 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.205969 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:20.209159 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:40145: BlockingRecv error: recv got EOF from 127.24.130.130:40145 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 02:51:20 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:20 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:20.423008 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:20.416757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:53903)
0806 02:51:20.417215 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:20.417233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:20.417389 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:20.417687 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:20.417690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:20.417956 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:20.418055 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:20.418061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.418926 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.418931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:20.419515 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:20.419524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.420011 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.420015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:20.420099 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:20.420682 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:20.420694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:20.422815 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":107,"threads_started":1}
W20260806 02:51:20.423094 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:20.416963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:53903 (local address 127.24.130.190:44967)
0806 02:51:20.417162 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:20.417167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:20.417329 (+ 162us) server_negotiation.cc:407] Connection header received
0806 02:51:20.417508 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:20.417511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:20.417568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:20.417664 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:20.418199 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.418812 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.419648 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:20.419881 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:20.419993 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:20.422973 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:53903: BlockingRecv error: recv got EOF from 127.24.130.131:53903 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 02:51:21 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:21 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:21.026892 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:21.021083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:35541)
0806 02:51:21.021430 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:21.021445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:21.021542 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:21.021839 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:21.021842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:21.022043 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:21.022148 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:21.022154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.022945 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.022948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:21.023526 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:21.023535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.023974 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.023976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:21.024040 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:21.024669 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:21.024682 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:21.026722 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":157,"threads_started":1}
W20260806 02:51:21.027014 5822 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:21.021298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:35541 (local address 127.24.130.190:44967)
0806 02:51:21.021562 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:21.021566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:21.021579 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:51:21.021618 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:21.021620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:21.021662 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:21.021748 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:21.022275 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.022841 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.023635 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.023868 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.023951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:21.026860 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:35541: BlockingRecv error: recv got EOF from 127.24.130.129:35541 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
Aug 06 02:51:21 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:21 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:21.216006 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:21.210011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:44637)
0806 02:51:21.210318 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:21.210334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:21.210454 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:21.210712 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:21.210714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:21.210942 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:21.211044 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:21.211050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.211998 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.212001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:21.212566 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:21.212575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.213033 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.213035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:21.213100 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:21.213775 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:21.213788 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:21.215869 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
W20260806 02:51:21.216240 5822 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:21.210172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:44637 (local address 127.24.130.190:44967)
0806 02:51:21.210355 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:21.210359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:21.210392 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:51:21.210546 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:21.210548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:21.210594 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:21.210693 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:21.211156 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.211848 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.212684 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.212922 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.213001 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:21.216102 (+ 3101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:44637: BlockingRecv error: recv got EOF from 127.24.130.130:44637 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 02:51:21 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:21 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:21.430311 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:21.424148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:49981)
0806 02:51:21.424492 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:21.424507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:21.424665 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:21.425034 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:21.425037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:21.425317 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:21.425437 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:21.425464 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.426392 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.426395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:21.427083 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:21.427093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.427540 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.427543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:21.427610 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:21.428201 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:21.428223 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:21.430184 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
W20260806 02:51:21.430519 5822 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:21.424230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:49981 (local address 127.24.130.190:44967)
0806 02:51:21.424392 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:21.424397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:21.424639 (+ 242us) server_negotiation.cc:407] Connection header received
0806 02:51:21.424793 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:21.424799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:21.424867 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:21.424976 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:21.425593 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.426280 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.427204 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:21.427421 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:21.427505 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:21.430348 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:49981: BlockingRecv error: recv got EOF from 127.24.130.131:49981 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:51:22 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:22 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:22.036614 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:22.027732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:47845)
0806 02:51:22.028067 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:22.028084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:22.028180 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:22.028454 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:22.028456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:22.028656 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:22.028757 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:22.028762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.029556 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.029559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:22.030126 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:22.030134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.030569 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.030571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:22.030635 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:22.034135 (+ 3500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:22.034149 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:22.036417 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
W20260806 02:51:22.036970 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:22.027860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47845 (local address 127.24.130.190:44967)
0806 02:51:22.028139 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:22.028142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:22.028157 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:22.028284 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:22.028286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:22.028332 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:22.028410 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:22.028899 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.029454 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.030231 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.030466 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.030545 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:22.036666 (+ 6121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:47845: BlockingRecv error: recv got EOF from 127.24.130.129:47845 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
Aug 06 02:51:22 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:22 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:22.223120 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:22.216938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:36517)
0806 02:51:22.217322 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:22.217379 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:22.217529 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:22.217833 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:22.217836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:22.218065 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:22.218164 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:22.218170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.219058 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.219063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:22.219637 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:22.219645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.220149 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.220152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:22.220242 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:22.220835 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:22.220847 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:22.222998 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":128,"threads_started":1}
W20260806 02:51:22.223335 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:22.216988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:36517 (local address 127.24.130.190:44967)
0806 02:51:22.217167 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:22.217172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:22.217474 (+ 302us) server_negotiation.cc:407] Connection header received
0806 02:51:22.217636 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:22.217639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:22.217692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:22.217799 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:22.218295 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.218940 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.219779 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.220019 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.220133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:22.223198 (+ 3065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:36517: BlockingRecv error: recv got EOF from 127.24.130.130:36517 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 02:51:22 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:22 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:22.438299 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:22.431298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:40469)
0806 02:51:22.431716 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:22.431739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:22.431933 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:22.432399 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:22.432402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:22.432719 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:22.432861 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:22.432868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.433946 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.433949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:22.434592 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:22.434601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.435202 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.435206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:22.435282 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:22.435926 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:22.435939 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:22.438144 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
W20260806 02:51:22.438582 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:22.431482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:40469 (local address 127.24.130.190:44967)
0806 02:51:22.431716 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:22.431723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:22.431853 (+ 130us) server_negotiation.cc:407] Connection header received
0806 02:51:22.432131 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:22.432138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:22.432206 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:22.432330 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:22.433018 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.433824 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.434819 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:22.435071 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:22.435174 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:22.438437 (+ 3263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:40469: BlockingRecv error: recv got EOF from 127.24.130.131:40469 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 06 02:51:23 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:23 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:23.043962 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:23.037570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:47875)
0806 02:51:23.037926 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:23.037944 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:23.038078 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:23.038388 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:23.038390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:23.038655 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:23.038815 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:23.038822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.039892 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.039898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:23.040484 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:23.040493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.040940 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.040943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:23.041005 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.041628 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:23.041645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:23.043838 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
W20260806 02:51:23.044153 5836 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:23.037718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47875 (local address 127.24.130.190:44967)
0806 02:51:23.038045 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:23.038049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:23.038065 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:51:23.038171 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:23.038173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:23.038218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:23.038354 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:23.038988 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.039754 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.040600 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.040828 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.040909 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.044005 (+ 3096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:47875: BlockingRecv error: recv got EOF from 127.24.130.129:47875 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":75,"threads_started":1}
Aug 06 02:51:23 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:23 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:23.229970 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:23.223974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:59545)
0806 02:51:23.224372 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:23.224387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:23.224522 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:23.224826 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:23.224829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:23.225060 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:23.225157 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:23.225162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.226000 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.226005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:23.226582 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:23.226589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.227123 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.227126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:23.227226 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.227797 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:23.227809 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:23.229846 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
W20260806 02:51:23.230301 5836 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:23.224116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:59545 (local address 127.24.130.190:44967)
0806 02:51:23.224313 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:23.224317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:23.224469 (+ 152us) server_negotiation.cc:407] Connection header received
0806 02:51:23.224639 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:23.224644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:23.224707 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:23.224800 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:23.225290 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.225878 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.226761 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.226998 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.227116 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.230020 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:59545: BlockingRecv error: recv got EOF from 127.24.130.130:59545 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 02:51:23 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:23 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:23.445052 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:23.439233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:43775)
0806 02:51:23.439542 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:23.439562 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:23.439688 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:23.439969 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:23.439971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:23.440166 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:23.440267 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:23.440272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.441017 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.441020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:23.441585 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:23.441593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.442066 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.442070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:23.442143 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.442840 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:23.442852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:23.444926 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
W20260806 02:51:23.445230 5836 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:23.439405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:43775 (local address 127.24.130.190:44967)
0806 02:51:23.439558 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:23.439566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:23.439646 (+ 80us) server_negotiation.cc:407] Connection header received
0806 02:51:23.439773 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:23.439775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:23.439856 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:23.439927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:23.440374 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.440915 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.441694 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.441924 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.442001 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.445093 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:43775: BlockingRecv error: recv got EOF from 127.24.130.131:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:51:24 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:24 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:24.050351 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:24.044836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:39455)
0806 02:51:24.045143 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:24.045159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:24.045259 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:24.045507 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:24.045509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:24.045706 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:24.045810 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:24.045816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.046605 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.046608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:24.047226 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:24.047236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.047669 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.047672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:24.047734 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:24.048286 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:24.048298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:24.050227 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20260806 02:51:24.050606 5842 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:24.045013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:39455 (local address 127.24.130.190:44967)
0806 02:51:24.045271 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:24.045276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:24.045294 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:51:24.045335 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:24.045337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:24.045378 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:24.045465 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:24.045940 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.046506 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.047341 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.047563 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.047657 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:24.050454 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:39455: BlockingRecv error: recv got EOF from 127.24.130.129:39455 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
Aug 06 02:51:24 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:24 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:24.236677 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:24.230917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:57863)
0806 02:51:24.231203 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:24.231242 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:24.231384 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:24.231717 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:24.231720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:24.231966 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:24.232071 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:24.232077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.232823 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.232826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:24.233390 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:24.233397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.233930 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.233932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:24.233997 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:24.234584 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:24.234595 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:24.236540 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
W20260806 02:51:24.236873 5842 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:24.231004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:57863 (local address 127.24.130.190:44967)
0806 02:51:24.231173 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:24.231178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:24.231332 (+ 154us) server_negotiation.cc:407] Connection header received
0806 02:51:24.231490 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:24.231493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:24.231555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:24.231643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:24.232184 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.232725 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.233549 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.233819 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.233914 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:24.236740 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:57863: BlockingRecv error: recv got EOF from 127.24.130.130:57863 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:51:24 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:24 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:24.452497 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:24.446030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:38383)
0806 02:51:24.446532 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:24.446547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:24.446753 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:24.447115 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:24.447117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:24.447395 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:24.447544 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:24.447555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.448412 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.448414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:24.448997 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:24.449006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.449480 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.449482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:24.449549 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:24.450227 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:24.450239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:24.452376 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":100,"threads_started":1}
W20260806 02:51:24.452706 5842 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:24.446195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38383 (local address 127.24.130.190:44967)
0806 02:51:24.446360 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:24.446366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:24.446852 (+ 486us) server_negotiation.cc:407] Connection header received
0806 02:51:24.446895 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:24.446897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:24.446962 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:24.447053 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:24.447668 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.448301 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.449128 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:24.449364 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:24.449464 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:24.452531 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:38383: BlockingRecv error: recv got EOF from 127.24.130.131:38383 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 02:51:25 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:25 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:25.056890 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:25.051305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:58621)
0806 02:51:25.051599 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:25.051616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:25.051723 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:25.051983 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:25.051985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:25.052182 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:25.052283 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:25.052288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.053074 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.053076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:25.053618 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:25.053625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.054050 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.054053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:25.054112 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.054643 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:25.054679 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:25.056679 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
W20260806 02:51:25.056967 5847 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:25.051390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:58621 (local address 127.24.130.190:44967)
0806 02:51:25.051681 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:25.051684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:25.051699 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:25.051812 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:25.051814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:25.051863 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:25.051945 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:25.052397 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.052972 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.053722 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.053947 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.054026 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.056834 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:58621: BlockingRecv error: recv got EOF from 127.24.130.129:58621 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
Aug 06 02:51:25 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:25 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:25.243708 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:25.237672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:42585)
0806 02:51:25.238050 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:25.238067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:25.238220 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:25.238550 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:25.238553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:25.238814 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:25.238928 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:25.238935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.239734 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.239740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:25.240298 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:25.240306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.240792 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.240795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:25.240874 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.241510 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:25.241522 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:25.243524 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":148,"threads_started":1}
W20260806 02:51:25.243795 5847 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:25.237799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:42585 (local address 127.24.130.190:44967)
0806 02:51:25.237980 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:25.237985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:25.238164 (+ 179us) server_negotiation.cc:407] Connection header received
0806 02:51:25.238344 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:25.238347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:25.238401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:25.238495 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:25.239075 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.239612 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.240432 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.240668 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.240786 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.243674 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:42585: BlockingRecv error: recv got EOF from 127.24.130.130:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 02:51:25 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:25 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:25.459838 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:25.453412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:59439)
0806 02:51:25.453918 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:25.453935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:25.454066 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:25.454398 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:25.454401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:25.454692 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:25.454817 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:25.454825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.455916 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.455918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:25.456478 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:25.456486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.456958 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.456960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:25.457021 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.457585 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:25.457596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:25.459716 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
W20260806 02:51:25.460090 5847 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:25.453656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:59439 (local address 127.24.130.190:44967)
0806 02:51:25.453806 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:25.453811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:25.454065 (+ 254us) server_negotiation.cc:407] Connection header received
0806 02:51:25.454144 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:25.454149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:25.454239 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:25.454354 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:25.454962 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.455802 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.456593 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.456850 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.456940 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.459939 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:59439: BlockingRecv error: recv got EOF from 127.24.130.131:59439 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 02:51:26 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:26 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:26.064193 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:26.058050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:34671)
0806 02:51:26.058434 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:26.058460 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:26.058580 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:26.058933 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:26.058936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:26.059208 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:26.059340 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:26.059348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.060274 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.060277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:26.060848 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:26.060856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.061283 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.061286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:26.061350 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:26.062008 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:26.062021 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:26.064071 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
W20260806 02:51:26.064433 5851 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:26.058187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:34671 (local address 127.24.130.190:44967)
0806 02:51:26.058485 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:26.058489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:26.058508 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:51:26.058714 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:26.058718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:26.058777 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:26.058879 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:26.059535 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.060147 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.060955 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.061180 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.061258 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:26.064279 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:34671: BlockingRecv error: recv got EOF from 127.24.130.129:34671 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
Aug 06 02:51:26 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:26 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:26.250197 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:26.244560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:50089)
0806 02:51:26.244859 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:26.244876 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:26.244971 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:26.245210 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:26.245212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:26.245412 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:26.245511 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:26.245517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.246261 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.246264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:26.246937 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:26.246947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.247370 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.247373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:26.247438 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:26.248097 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:26.248109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:26.250061 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
W20260806 02:51:26.250363 5851 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:26.244757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:50089 (local address 127.24.130.190:44967)
0806 02:51:26.244929 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:26.244933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:26.244946 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:51:26.245042 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:26.245044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:26.245087 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:26.245159 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:26.245620 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.246148 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.247049 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.247265 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.247343 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:26.250223 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:50089: BlockingRecv error: recv got EOF from 127.24.130.130:50089 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 02:51:26 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:26 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:26.466910 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:26.460642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:43283)
0806 02:51:26.461031 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:26.461045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:26.461147 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:26.461472 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:26.461475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:26.461716 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:26.461841 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:26.461847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.462681 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.462685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:26.463266 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:26.463276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.463862 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.463865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:26.463930 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:26.464464 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:26.464477 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:26.466647 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
W20260806 02:51:26.466953 5851 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:26.460774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:43283 (local address 127.24.130.190:44967)
0806 02:51:26.460947 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:26.460952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:26.461118 (+ 166us) server_negotiation.cc:407] Connection header received
0806 02:51:26.461239 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:26.461241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:26.461292 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:26.461367 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:26.461984 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.462532 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.463403 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:26.463733 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:26.463826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:26.466838 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:43283: BlockingRecv error: recv got EOF from 127.24.130.131:43283 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 02:51:27 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:27 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:27.075980 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:27.065255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:52299)
0806 02:51:27.065576 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:27.065603 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:27.065712 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:27.066018 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:27.066021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:27.066306 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:27.066467 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:27.066480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.067452 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.067455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:27.068102 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:27.068111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.068597 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.068600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:27.068667 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:27.073547 (+ 4880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:27.073562 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:27.075830 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
W20260806 02:51:27.076195 5857 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:27.065368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:52299 (local address 127.24.130.190:44967)
0806 02:51:27.065697 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:27.065702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:27.065718 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:51:27.065792 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:27.065795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:27.065852 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:27.065950 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:27.066607 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.067339 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.068238 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.068481 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.068580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:27.076012 (+ 7432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:52299: BlockingRecv error: recv got EOF from 127.24.130.129:52299 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
Aug 06 02:51:27 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:27 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:27.257843 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:27.251466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:50713)
0806 02:51:27.251789 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:27.251806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:27.251951 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:27.252214 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:27.252216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:27.252415 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:27.252511 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:27.252517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.253325 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.253328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:27.253935 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:27.253945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.254465 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.254467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:27.254537 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:27.255205 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:27.255217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:27.257720 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
W20260806 02:51:27.257972 5857 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:27.251556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:50713 (local address 127.24.130.190:44967)
0806 02:51:27.251770 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:27.251773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:27.251939 (+ 166us) server_negotiation.cc:407] Connection header received
0806 02:51:27.252046 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:27.252049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:27.252100 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:27.252169 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:27.252652 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.253203 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.254077 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.254333 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.254428 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:27.257840 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:50713: BlockingRecv error: recv got EOF from 127.24.130.130:50713 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 02:51:27 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:27 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.131@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:27.474521 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:27.467983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.131:41535)
0806 02:51:27.468342 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:27.468361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:27.468487 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:27.468878 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:27.468880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:27.469107 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:27.469234 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:27.469239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.470034 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.470038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:27.470633 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:27.470642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.471199 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.471202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:27.471271 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:27.471901 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:27.471913 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:27.474339 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
W20260806 02:51:27.474785 5857 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:27.468191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:41535 (local address 127.24.130.190:44967)
0806 02:51:27.468370 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:27.468374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:27.468445 (+ 71us) server_negotiation.cc:407] Connection header received
0806 02:51:27.468663 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:27.468666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:27.468720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:27.468812 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:27.469367 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.469906 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.470870 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:27.471088 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:27.471169 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:27.474521 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.131:41535: BlockingRecv error: recv got EOF from 127.24.130.131:41535 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 02:51:28 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:28 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.129@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:28.083091 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:28.077068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.129:52449)
0806 02:51:28.077428 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:28.077447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:28.077592 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:28.077908 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:28.077911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:28.078147 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:28.078245 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:28.078252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.079126 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.079130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:28.079699 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:28.079707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.080184 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.080188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:28.080270 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.080863 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:28.080875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:28.082968 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
W20260806 02:51:28.083316 5863 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:28.077128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:52449 (local address 127.24.130.190:44967)
0806 02:51:28.077447 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:28.077451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:28.077535 (+ 84us) server_negotiation.cc:407] Connection header received
0806 02:51:28.077704 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:28.077707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:28.077789 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:28.077887 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:28.078381 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.079008 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.079827 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.080061 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.080177 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.083146 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.129:52449: BlockingRecv error: recv got EOF from 127.24.130.129:52449 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":125,"threads_started":1}
Aug 06 02:51:28 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:51:28 dist-test-slave-3kk6 krb5kdc[4333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.130.130@KRBTEST.COM for oryx/127.24.130.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:51:28.264928 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:28.258896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44967 (local address 127.24.130.130:49193)
0806 02:51:28.259240 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:28.259263 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:28.259395 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:28.259689 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:28.259692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:28.259936 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:28.260033 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:28.260038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.260853 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.260860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:28.261428 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:28.261437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.262028 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.262032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:28.262123 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.262756 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:28.262769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:28.264802 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.130.190:44967: Server oryx/127.24.130.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":98,"threads_started":1}
W20260806 02:51:28.265251 5863 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:28.259034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:49193 (local address 127.24.130.190:44967)
0806 02:51:28.259205 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:28.259210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:28.259343 (+ 133us) server_negotiation.cc:407] Connection header received
0806 02:51:28.259507 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:28.259511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:28.259568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:28.259660 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:28.260162 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.260715 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.261557 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.261901 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.262022 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.264979 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.130.130:49193: BlockingRecv error: recv got EOF from 127.24.130.130:49193 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:28.351706 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 5175
I20260806 02:51:28.365583 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 5396
I20260806 02:51:28.380923 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 5588
I20260806 02:51:28.401664 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 5075
2026-08-06T02:51:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18357 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T02:51:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:28Z Disabled control of system clock
I20260806 02:51:28.441387 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:35267
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:51:28.560503 5870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:28.560813 5870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:28.560849 5870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:28.565043 5870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:28.565129 5870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:28.565145 5870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:28.565164 5870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:28.565181 5870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:28.570341 5870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:35267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:28.571637 5870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:28.573189 5870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:51:28.582942 5870 server_base.cc:1034] running on GCE node
W20260806 02:51:28.582898 5876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:28.582898 5877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:28.582901 5881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:28.583827 5870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:28.585032 5870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:28.586265 5870 hybrid_clock.cc:648] HybridClock initialized: now 1785984688586248 us; error 33 us; skew 500 ppm
I20260806 02:51:28.588442 5870 webserver.cc:460] Webserver started at http://127.24.130.190:42853/ using document root <none> and password file <none>
I20260806 02:51:28.589080 5870 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:28.589215 5870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:28.589493 5870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:28.591910 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "0d9dc00b50db4a32be4e9de2f0682cd0"
format_stamp: "Formatted at 2026-08-06 02:51:28 on dist-test-slave-3kk6"
server_key: "af7debf0c5081aace0360eb501feabbf"
server_key_iv: "0ad9e9c44bdf60782c096fa87c6a6b8d"
server_key_version: "encryptionkey@0"
I20260806 02:51:28.592572 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "0d9dc00b50db4a32be4e9de2f0682cd0"
format_stamp: "Formatted at 2026-08-06 02:51:28 on dist-test-slave-3kk6"
server_key: "af7debf0c5081aace0360eb501feabbf"
server_key_iv: "0ad9e9c44bdf60782c096fa87c6a6b8d"
server_key_version: "encryptionkey@0"
I20260806 02:51:28.596896 5870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:28.599783 5890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:28.601325 5870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:28.601538 5870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "0d9dc00b50db4a32be4e9de2f0682cd0"
format_stamp: "Formatted at 2026-08-06 02:51:28 on dist-test-slave-3kk6"
server_key: "af7debf0c5081aace0360eb501feabbf"
server_key_iv: "0ad9e9c44bdf60782c096fa87c6a6b8d"
server_key_version: "encryptionkey@0"
I20260806 02:51:28.601692 5870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:28.614614 5870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:28.615545 5870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:28.615782 5870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:28.622905 5870 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:35267
I20260806 02:51:28.622931 5971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:35267 every 8 connection(s)
I20260806 02:51:28.623922 5870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:28.627394 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:28.628912 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 5870
I20260806 02:51:28.629066 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:51:28.629382 25098 external_mini_cluster.cc:1428] Setting key 8557c1daef223086ca1c249f2bd48195
I20260806 02:51:28.634912 5973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0: Bootstrap starting.
I20260806 02:51:28.637648 5973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:28.638644 5973 log.cc:824] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:28.640764 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:28.631099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.24.130.190:35267)
0806 02:51:28.631885 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:28.631908 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:28.631954 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:51:28.632890 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:28.632916 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:28.633440 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:28.633853 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:28.634306 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.635794 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.636747 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.637450 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.637636 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.638601 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:28.638608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:28.638624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:28.639195 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:28.639254 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:28.639275 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:28.639356 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:28.639534 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 02:51:28.639865 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":118,"threads_started":1}
I20260806 02:51:28.641111 5973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0: No bootstrap required, opened a new log
I20260806 02:51:28.645104 5973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35267 } }
I20260806 02:51:28.645362 5973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:28.645398 5973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9dc00b50db4a32be4e9de2f0682cd0, State: Initialized, Role: FOLLOWER
I20260806 02:51:28.645828 5973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35267 } }
I20260806 02:51:28.645959 5973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:28.646011 5973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:28.646090 5973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:28.647154 5973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35267 } }
I20260806 02:51:28.647550 5973 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d9dc00b50db4a32be4e9de2f0682cd0; no voters:
I20260806 02:51:28.648000 5973 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:28.648083 5981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:28.648315 5981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9dc00b50db4a32be4e9de2f0682cd0, State: Running, Role: LEADER
I20260806 02:51:28.648691 5981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35267 } }
I20260806 02:51:28.649430 5973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:28.650548 5983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9dc00b50db4a32be4e9de2f0682cd0. Latest consensus state: current_term: 1 leader_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35267 } } }
I20260806 02:51:28.650724 5983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:28.651129 5992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:28.651103 5982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9dc00b50db4a32be4e9de2f0682cd0" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 35267 } } }
I20260806 02:51:28.651262 5982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:28.654719 5992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:28.660735 5992 catalog_manager.cc:1347] Generated new cluster ID: 584432234b4d4d9d85bf139a9397747b
I20260806 02:51:28.660832 5992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:28.671375 5992 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:28.672729 5992 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:28.679021 5992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d9dc00b50db4a32be4e9de2f0682cd0: Generated new TSK 0
I20260806 02:51:28.679896 5992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:28.689971 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:35267
--builtin_ntp_servers=127.24.130.148:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 02:51:28.810493 6007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:28.810990 6007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:28.811082 6007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:28.815275 6007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:28.815373 6007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:28.815469 6007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:28.820410 6007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:35267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:28.821619 6007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:28.823096 6007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:28.833621 6016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:28.833621 6014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:28.833637 6013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:28.833935 6007 server_base.cc:1034] running on GCE node
I20260806 02:51:28.834512 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:28.835292 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:28.836526 6007 hybrid_clock.cc:648] HybridClock initialized: now 1785984688836500 us; error 52 us; skew 500 ppm
I20260806 02:51:28.838624 6007 webserver.cc:460] Webserver started at http://127.24.130.129:45847/ using document root <none> and password file <none>
I20260806 02:51:28.839315 6007 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:28.839419 6007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:28.839670 6007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:28.841930 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "c611c67d518b4e65a46db46086379e76"
format_stamp: "Formatted at 2026-08-06 02:51:28 on dist-test-slave-3kk6"
server_key: "173c596957730b073f0880acaa85ec64"
server_key_iv: "c429e11ebca48fb1168ed98c2d183263"
server_key_version: "encryptionkey@0"
I20260806 02:51:28.842564 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "c611c67d518b4e65a46db46086379e76"
format_stamp: "Formatted at 2026-08-06 02:51:28 on dist-test-slave-3kk6"
server_key: "173c596957730b073f0880acaa85ec64"
server_key_iv: "c429e11ebca48fb1168ed98c2d183263"
server_key_version: "encryptionkey@0"
I20260806 02:51:28.846742 6007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:51:28.849522 6025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:28.851089 6007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:28.851248 6007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "c611c67d518b4e65a46db46086379e76"
format_stamp: "Formatted at 2026-08-06 02:51:28 on dist-test-slave-3kk6"
server_key: "173c596957730b073f0880acaa85ec64"
server_key_iv: "c429e11ebca48fb1168ed98c2d183263"
server_key_version: "encryptionkey@0"
I20260806 02:51:28.851384 6007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:28.870364 6007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:28.871274 6007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:28.871498 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:28.872180 6007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:28.873239 6007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:28.873314 6007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:28.873405 6007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:28.873454 6007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:28.883222 6007 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:44785
I20260806 02:51:28.883251 6203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:44785 every 8 connection(s)
I20260806 02:51:28.884268 6007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:28.888587 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 6007
I20260806 02:51:28.888697 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:28.889004 25098 external_mini_cluster.cc:1428] Setting key 3d1673437d59212d1522aa8680afc64e
I20260806 02:51:28.891376 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:35267
--builtin_ntp_servers=127.24.130.148:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:51:28.895053 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:28.886252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:54613 (local address 127.24.130.190:35267)
0806 02:51:28.886415 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:28.886419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:28.887565 (+ 1146us) server_negotiation.cc:407] Connection header received
0806 02:51:28.889096 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:28.889100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:28.889166 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:28.889291 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:28.890393 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.891205 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.892477 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.892894 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.893026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.894465 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:28.894469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:28.894472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:28.894487 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:28.894510 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:28.894520 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:28.894598 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:28.894838 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:51:28.894906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:51:28.896313 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:28.886639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:35267 (local address 127.24.130.129:54613)
0806 02:51:28.887402 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:28.887446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:28.888859 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:28.889475 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:28.889483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:28.889571 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:28.890225 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:28.890245 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.891351 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.891354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:28.892345 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:28.892355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:28.893057 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:28.893060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:28.893331 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:28.894130 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:51:28.894134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:28.894195 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:51:28.894289 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:28.894620 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:28.894632 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:28.894639 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:51:28.894841 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 02:51:28.895176 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260806 02:51:28.897830 6204 heartbeater.cc:347] Connected to a master server at 127.24.130.190:35267
I20260806 02:51:28.898119 6204 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:28.898819 6204 heartbeater.cc:511] Master 127.24.130.190:35267 requested a full tablet report, sending...
I20260806 02:51:28.900545 5915 ts_manager.cc:194] Registered new tserver with Master: c611c67d518b4e65a46db46086379e76 (127.24.130.129:44785)
I20260806 02:51:28.901754 5915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.129:54613
W20260806 02:51:29.013949 6207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:29.014271 6207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:29.014319 6207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:29.018582 6207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:29.018721 6207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:29.018779 6207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:29.023896 6207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:35267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:29.025144 6207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:29.026628 6207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:29.037303 6218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.037352 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.037616 6207 server_base.cc:1034] running on GCE node
W20260806 02:51:29.037352 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.038153 6207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:29.039029 6207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:29.040267 6207 hybrid_clock.cc:648] HybridClock initialized: now 1785984689040247 us; error 36 us; skew 500 ppm
I20260806 02:51:29.042490 6207 webserver.cc:460] Webserver started at http://127.24.130.130:34097/ using document root <none> and password file <none>
I20260806 02:51:29.043226 6207 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:29.043337 6207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:29.043601 6207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:29.045902 6207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "c7219a76dfc2400d84c8ae57e649ddea"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "e5935c7a592b94c1098f397bc59335ca"
server_key_iv: "ea6e430cc35cbbd1afa3ac9e639fac5c"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.046551 6207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "c7219a76dfc2400d84c8ae57e649ddea"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "e5935c7a592b94c1098f397bc59335ca"
server_key_iv: "ea6e430cc35cbbd1afa3ac9e639fac5c"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.050997 6207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:51:29.053931 6227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.055383 6207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:29.055541 6207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "c7219a76dfc2400d84c8ae57e649ddea"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "e5935c7a592b94c1098f397bc59335ca"
server_key_iv: "ea6e430cc35cbbd1afa3ac9e639fac5c"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.055699 6207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:29.074438 6207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:29.075302 6207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:29.075484 6207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:29.076081 6207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:29.077049 6207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:29.077097 6207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.077137 6207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:29.077163 6207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.086761 6207 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:45647
I20260806 02:51:29.086776 6382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:45647 every 8 connection(s)
I20260806 02:51:29.087777 6207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:29.090948 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 6207
I20260806 02:51:29.091094 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:29.091423 25098 external_mini_cluster.cc:1428] Setting key cfb976507301beeb23a51351efb91fe0
I20260806 02:51:29.093734 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:35267
--builtin_ntp_servers=127.24.130.148:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:51:29.099363 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.089677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:47765 (local address 127.24.130.190:35267)
0806 02:51:29.089830 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:29.089834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:29.091064 (+ 1230us) server_negotiation.cc:407] Connection header received
0806 02:51:29.092852 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:29.092858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:29.092921 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:29.093004 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:29.094116 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.095020 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.096800 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.097026 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.097115 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:29.098646 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:29.098711 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:29.098716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:29.098737 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:29.098787 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:29.098798 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:29.098866 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:29.099127 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:51:29.099210 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:51:29.100757 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.090103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:35267 (local address 127.24.130.130:47765)
0806 02:51:29.090875 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:29.090924 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:29.092554 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:29.093237 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:29.093245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:29.093345 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:29.093951 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:29.093980 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.095202 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.095205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:29.096625 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:29.096639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.097212 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.097219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:29.097471 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:29.098282 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:51:29.098285 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:29.098350 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:51:29.098455 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:29.098904 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:29.098918 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:29.098928 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:51:29.099107 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 02:51:29.099496 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260806 02:51:29.102151 6383 heartbeater.cc:347] Connected to a master server at 127.24.130.190:35267
I20260806 02:51:29.102453 6383 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:29.103268 6383 heartbeater.cc:511] Master 127.24.130.190:35267 requested a full tablet report, sending...
I20260806 02:51:29.104660 5915 ts_manager.cc:194] Registered new tserver with Master: c7219a76dfc2400d84c8ae57e649ddea (127.24.130.130:45647)
I20260806 02:51:29.105582 5915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.130:47765
W20260806 02:51:29.219041 6386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:29.219362 6386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:29.219400 6386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:29.223728 6386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:29.223811 6386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:29.223869 6386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:29.228826 6386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:35267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:29.230072 6386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:29.231587 6386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:51:29.242101 6386 server_base.cc:1034] running on GCE node
W20260806 02:51:29.242049 6400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.242034 6397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.242071 6398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.242951 6386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:29.243665 6386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:29.244866 6386 hybrid_clock.cc:648] HybridClock initialized: now 1785984689244850 us; error 30 us; skew 500 ppm
I20260806 02:51:29.246920 6386 webserver.cc:460] Webserver started at http://127.24.130.131:43481/ using document root <none> and password file <none>
I20260806 02:51:29.247547 6386 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:29.247651 6386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:29.247905 6386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:29.250290 6386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "b68c9cf9fc9642b1a1279c47a9fff4a2"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "a7b0430ac71a74446b53f1fa5bf96d10"
server_key_iv: "606500d75bf36c5261a99ff6d18c14b0"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.250962 6386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "b68c9cf9fc9642b1a1279c47a9fff4a2"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "a7b0430ac71a74446b53f1fa5bf96d10"
server_key_iv: "606500d75bf36c5261a99ff6d18c14b0"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.255061 6386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:51:29.257890 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.259305 6386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:29.259464 6386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "b68c9cf9fc9642b1a1279c47a9fff4a2"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "a7b0430ac71a74446b53f1fa5bf96d10"
server_key_iv: "606500d75bf36c5261a99ff6d18c14b0"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.259598 6386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:29.273140 6386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:29.273957 6386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:29.274189 6386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:29.274948 6386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:29.276003 6386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:29.276079 6386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.276157 6386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:29.276211 6386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.286262 6386 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:42389
I20260806 02:51:29.286279 6564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:42389 every 8 connection(s)
I20260806 02:51:29.287860 6386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:29.293087 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 6386
I20260806 02:51:29.293234 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:29.293576 25098 external_mini_cluster.cc:1428] Setting key 8d9a6920ed305e6e4179dbd071d3473a
I20260806 02:51:29.300813 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.290354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:48439 (local address 127.24.130.190:35267)
0806 02:51:29.290546 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:29.290553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:29.291984 (+ 1431us) server_negotiation.cc:407] Connection header received
0806 02:51:29.294292 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:29.294299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:29.294384 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:29.294506 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:29.295750 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.296517 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.298127 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.298456 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.298578 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:29.300109 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:29.300115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:29.300120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:29.300139 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:29.300169 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:29.300184 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:29.300278 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:29.300500 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:51:29.300584 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:51:29.301843 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.290802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:35267 (local address 127.24.130.131:48439)
0806 02:51:29.291779 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:29.291849 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:29.293914 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:29.294754 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:29.294765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:29.294879 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:29.295569 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:29.295589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.296693 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.296697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:29.297986 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:29.297998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:29.298641 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:29.298644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:29.298932 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:29.299732 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:51:29.299734 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:29.299802 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:51:29.299912 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:29.300312 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:29.300321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:29.300327 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:51:29.300454 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:51:29.300726 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":178,"threads_started":1}
I20260806 02:51:29.303414 6565 heartbeater.cc:347] Connected to a master server at 127.24.130.190:35267
I20260806 02:51:29.303761 6565 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:29.304598 6565 heartbeater.cc:511] Master 127.24.130.190:35267 requested a full tablet report, sending...
I20260806 02:51:29.307477 5915 ts_manager.cc:194] Registered new tserver with Master: b68c9cf9fc9642b1a1279c47a9fff4a2 (127.24.130.131:42389)
I20260806 02:51:29.308116 5915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.131:48439
I20260806 02:51:29.310531 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:51:29.314144 5979 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:29.313034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.24.130.190:35267)
0806 02:51:29.313179 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:29.313185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:29.313389 (+ 204us) server_negotiation.cc:407] Connection header received
0806 02:51:29.313564 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:29.313567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:29.313626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:29.313714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:29.313995 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52272: BlockingRecv error: recv got EOF from 127.0.0.1:52272 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:51:29.314270 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 6007
I20260806 02:51:29.326826 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 6207
I20260806 02:51:29.340931 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 6386
I20260806 02:51:29.358037 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 5870
2026-08-06T02:51:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (959 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T02:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:29Z Disabled control of system clock
I20260806 02:51:29.401510 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:34735
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:34735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:51:29.520804 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:29.521126 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:29.521166 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:29.525383 6577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:29.525477 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:29.525493 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:29.525511 6577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:29.525527 6577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:29.530555 6577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:34735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:29.531860 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:29.533427 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:29.543267 6589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.543270 6586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.543268 6587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.543625 6577 server_base.cc:1034] running on GCE node
I20260806 02:51:29.544297 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:29.545588 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:29.546756 6577 hybrid_clock.cc:648] HybridClock initialized: now 1785984689546745 us; error 46 us; skew 500 ppm
I20260806 02:51:29.549782 6577 webserver.cc:460] Webserver started at http://127.24.130.190:41445/ using document root <none> and password file <none>
I20260806 02:51:29.550485 6577 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:29.550563 6577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:29.550853 6577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:29.553292 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "7ea1517b51d340d594578b2e0eccd143"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "ea4d26a12ab9ad2ca9616632fb21152a"
server_key_iv: "98caf6e29206f382b8645071b947b5f4"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.553988 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "7ea1517b51d340d594578b2e0eccd143"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "ea4d26a12ab9ad2ca9616632fb21152a"
server_key_iv: "98caf6e29206f382b8645071b947b5f4"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.558460 6577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:51:29.561631 6597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.563287 6577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:29.563488 6577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "7ea1517b51d340d594578b2e0eccd143"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "ea4d26a12ab9ad2ca9616632fb21152a"
server_key_iv: "98caf6e29206f382b8645071b947b5f4"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.563649 6577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:29.569505 6577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:29.570312 6577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:29.570552 6577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:29.578523 6577 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:34735
I20260806 02:51:29.578547 6673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:34735 every 8 connection(s)
I20260806 02:51:29.579778 6577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:29.583449 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:29.589433 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 6577
I20260806 02:51:29.589637 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:51:29.589895 6674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143: Bootstrap starting.
I20260806 02:51:29.589953 25098 external_mini_cluster.cc:1428] Setting key c0670c8b00938706834b4c18d10b3f00
I20260806 02:51:29.592954 6674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:29.593957 6674 log.cc:824] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:29.596485 6674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143: No bootstrap required, opened a new log
I20260806 02:51:29.596758 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.591579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.24.130.190:34735)
0806 02:51:29.592353 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:29.592378 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:29.592426 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:51:29.593451 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:29.593477 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:29.593970 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:29.594382 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:29.595327 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:29.595332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:29.595344 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:29.595354 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:29.595410 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:29.595427 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:29.595523 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:29.595658 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:51:29.595977 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":144,"threads_started":1}
I20260806 02:51:29.600330 6674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea1517b51d340d594578b2e0eccd143" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34735 } }
I20260806 02:51:29.600625 6674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:29.600729 6674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea1517b51d340d594578b2e0eccd143, State: Initialized, Role: FOLLOWER
I20260806 02:51:29.601202 6674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea1517b51d340d594578b2e0eccd143" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34735 } }
I20260806 02:51:29.601368 6674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:29.601464 6674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:29.601604 6674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:29.602753 6674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea1517b51d340d594578b2e0eccd143" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34735 } }
I20260806 02:51:29.603138 6674 leader_election.cc:302] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ea1517b51d340d594578b2e0eccd143; no voters:
I20260806 02:51:29.603543 6674 leader_election.cc:288] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:29.603690 6682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:29.604242 6682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [term 1 LEADER]: Becoming Leader. State: Replica: 7ea1517b51d340d594578b2e0eccd143, State: Running, Role: LEADER
I20260806 02:51:29.604642 6682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea1517b51d340d594578b2e0eccd143" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34735 } }
I20260806 02:51:29.605018 6674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:29.606303 6684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ea1517b51d340d594578b2e0eccd143. Latest consensus state: current_term: 1 leader_uuid: "7ea1517b51d340d594578b2e0eccd143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea1517b51d340d594578b2e0eccd143" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34735 } } }
I20260806 02:51:29.606441 6684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:29.606802 6683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ea1517b51d340d594578b2e0eccd143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ea1517b51d340d594578b2e0eccd143" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34735 } } }
I20260806 02:51:29.606889 6683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:29.606927 6693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:29.609921 6693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:29.616024 6693 catalog_manager.cc:1347] Generated new cluster ID: 2f2cd946428f4a3b9caab9521a242c67
I20260806 02:51:29.616147 6693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:29.631642 6693 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:29.632118 6693 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:29.639972 6693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ea1517b51d340d594578b2e0eccd143: Generated new TSK 0
I20260806 02:51:29.640977 6693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:29.656152 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:34735
--builtin_ntp_servers=127.24.130.148:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:51:29.779116 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:29.779443 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:29.779486 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:29.783824 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:29.783901 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:29.783963 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:29.788985 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:29.790299 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:29.791889 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:29.802068 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.802121 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.802332 6708 server_base.cc:1034] running on GCE node
W20260806 02:51:29.802069 6723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.802965 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:29.803745 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:29.805032 6708 hybrid_clock.cc:648] HybridClock initialized: now 1785984689804995 us; error 48 us; skew 500 ppm
I20260806 02:51:29.807452 6708 webserver.cc:460] Webserver started at http://127.24.130.129:42501/ using document root <none> and password file <none>
I20260806 02:51:29.808111 6708 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:29.808218 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:29.808457 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:29.810834 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "48b47428cde446409f8c23f373d93dd2"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "6273747f11f8c9579ef8bfe44fd5f7ba"
server_key_iv: "846811d958fbf8d999c906457a1de5b7"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.811467 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "48b47428cde446409f8c23f373d93dd2"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "6273747f11f8c9579ef8bfe44fd5f7ba"
server_key_iv: "846811d958fbf8d999c906457a1de5b7"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.815584 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:51:29.818427 6735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.819867 6708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:51:29.820025 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "48b47428cde446409f8c23f373d93dd2"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "6273747f11f8c9579ef8bfe44fd5f7ba"
server_key_iv: "846811d958fbf8d999c906457a1de5b7"
server_key_version: "encryptionkey@0"
I20260806 02:51:29.820171 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:29.825489 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:29.826176 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:29.826414 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:29.827142 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:29.828166 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:29.828239 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.828313 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:29.828362 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:29.838017 6708 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:44675
I20260806 02:51:29.838044 6895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:44675 every 8 connection(s)
I20260806 02:51:29.839138 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:29.845093 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 6708
I20260806 02:51:29.845239 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:29.845549 25098 external_mini_cluster.cc:1428] Setting key 48595e553bd2e37db4d295ce65ffdd90
I20260806 02:51:29.845849 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.841205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:51645 (local address 127.24.130.190:34735)
0806 02:51:29.841372 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:29.841376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:29.842469 (+ 1093us) server_negotiation.cc:407] Connection header received
0806 02:51:29.843902 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:29.843905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:29.843946 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:29.844026 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:29.845304 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:29.845307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:29.845310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:29.845328 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:29.845359 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:29.845368 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:29.845482 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:29.845634 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:51:29.845683 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:51:29.846911 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:29.841619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:34735 (local address 127.24.130.129:51645)
0806 02:51:29.842328 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:29.842372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:29.843694 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:29.844201 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:29.844208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:29.844308 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:29.845014 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:51:29.845015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:29.845056 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:51:29.845128 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:29.845461 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:29.845472 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:29.845478 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:51:29.845679 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 02:51:29.845937 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
I20260806 02:51:29.847680 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:34735
--builtin_ntp_servers=127.24.130.148:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:51:29.848172 6896 heartbeater.cc:347] Connected to a master server at 127.24.130.190:34735
I20260806 02:51:29.848434 6896 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:29.848937 6896 heartbeater.cc:511] Master 127.24.130.190:34735 requested a full tablet report, sending...
I20260806 02:51:29.850855 6619 ts_manager.cc:194] Registered new tserver with Master: 48b47428cde446409f8c23f373d93dd2 (127.24.130.129:44675)
W20260806 02:51:29.972460 6901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:29.972859 6901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:29.972944 6901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:29.977432 6901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:29.977542 6901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:29.977640 6901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:29.983490 6901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:29.984927 6901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:29.986809 6901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:29.997592 6911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.997632 6907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.998096 6901 server_base.cc:1034] running on GCE node
W20260806 02:51:29.997592 6908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.998965 6901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:29.999722 6901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:30.000931 6901 hybrid_clock.cc:648] HybridClock initialized: now 1785984690000917 us; error 35 us; skew 500 ppm
I20260806 02:51:30.003100 6901 webserver.cc:460] Webserver started at http://127.24.130.130:37261/ using document root <none> and password file <none>
I20260806 02:51:30.003726 6901 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:30.003841 6901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:30.004096 6901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:30.006735 6901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "1782f412036e492d8c81c9d24f6115f9"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "8270e8fcf14bd96329292a30fdfa4db4"
server_key_iv: "c55fd382aa6df6500ccbbd690329e016"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.007396 6901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "1782f412036e492d8c81c9d24f6115f9"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "8270e8fcf14bd96329292a30fdfa4db4"
server_key_iv: "c55fd382aa6df6500ccbbd690329e016"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.011514 6901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:51:30.014371 6920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.015817 6901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:30.015978 6901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "1782f412036e492d8c81c9d24f6115f9"
format_stamp: "Formatted at 2026-08-06 02:51:29 on dist-test-slave-3kk6"
server_key: "8270e8fcf14bd96329292a30fdfa4db4"
server_key_iv: "c55fd382aa6df6500ccbbd690329e016"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.016119 6901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:30.021047 6901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:30.021782 6901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:30.022001 6901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:30.022611 6901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:30.023723 6901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:30.023803 6901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.023874 6901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:30.023926 6901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.033540 6901 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:36113
I20260806 02:51:30.033565 7074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:36113 every 8 connection(s)
I20260806 02:51:30.034565 6901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:30.036993 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 6901
I20260806 02:51:30.037158 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:30.037492 25098 external_mini_cluster.cc:1428] Setting key a85ac2d6db61f3490303001ad7d0679e
I20260806 02:51:30.040014 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:34735
--builtin_ntp_servers=127.24.130.148:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:51:30.042274 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.036808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:41815 (local address 127.24.130.190:34735)
0806 02:51:30.037024 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.037029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.038312 (+ 1283us) server_negotiation.cc:407] Connection header received
0806 02:51:30.040179 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.040184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.040239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.040360 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.041709 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:30.041750 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:30.041753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:30.041774 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:30.041798 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:30.041807 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:30.041921 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:30.042070 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 02:51:30.042125 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:51:30.043363 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.037264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:34735 (local address 127.24.130.130:41815)
0806 02:51:30.038094 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.038183 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:30.039880 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:30.040565 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:30.040573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:30.040683 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:30.041395 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:51:30.041397 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:30.041444 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:51:30.041523 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:30.041899 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:30.041910 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:30.041916 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:51:30.042126 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 02:51:30.042423 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260806 02:51:30.044883 7077 heartbeater.cc:347] Connected to a master server at 127.24.130.190:34735
I20260806 02:51:30.045184 7077 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:30.045693 7077 heartbeater.cc:511] Master 127.24.130.190:34735 requested a full tablet report, sending...
I20260806 02:51:30.047032 6619 ts_manager.cc:194] Registered new tserver with Master: 1782f412036e492d8c81c9d24f6115f9 (127.24.130.130:36113)
W20260806 02:51:30.174417 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:30.174847 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:30.174928 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:30.179188 7082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:30.179289 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:30.179391 7082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:30.184566 7082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:34735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:30.185873 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:30.187494 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:30.198410 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:30.198426 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:30.198694 7082 server_base.cc:1034] running on GCE node
W20260806 02:51:30.198410 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:30.199398 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:30.200152 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:30.201336 7082 hybrid_clock.cc:648] HybridClock initialized: now 1785984690201320 us; error 30 us; skew 500 ppm
I20260806 02:51:30.203611 7082 webserver.cc:460] Webserver started at http://127.24.130.131:42977/ using document root <none> and password file <none>
I20260806 02:51:30.204274 7082 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:30.204345 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:30.204618 7082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:30.207228 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "37f84f28a39e4c4bac67d0d153d7277a"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-3kk6"
server_key: "40be608f2526dd5e5003add5cabfad94"
server_key_iv: "8887794e4855c766183d43ef466a710c"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.207942 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "37f84f28a39e4c4bac67d0d153d7277a"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-3kk6"
server_key: "40be608f2526dd5e5003add5cabfad94"
server_key_iv: "8887794e4855c766183d43ef466a710c"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.212565 7082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:51:30.215576 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.217111 7082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:30.217301 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "37f84f28a39e4c4bac67d0d153d7277a"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-3kk6"
server_key: "40be608f2526dd5e5003add5cabfad94"
server_key_iv: "8887794e4855c766183d43ef466a710c"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.217466 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:30.223079 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:30.223830 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:30.224046 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:30.224766 7082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:30.225868 7082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:30.225945 7082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.226022 7082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:30.226078 7082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.235513 7082 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:32981
I20260806 02:51:30.235577 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:32981 every 8 connection(s)
I20260806 02:51:30.236615 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:30.240595 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 7082
I20260806 02:51:30.240738 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:30.241074 25098 external_mini_cluster.cc:1428] Setting key 6a944aa50f0cf7747a2987ffe09587be
I20260806 02:51:30.244156 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.238819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:35229 (local address 127.24.130.190:34735)
0806 02:51:30.238982 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.238986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.240236 (+ 1250us) server_negotiation.cc:407] Connection header received
0806 02:51:30.242104 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.242110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.242157 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.242246 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.243698 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:30.243701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:30.243704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:30.243727 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:30.243752 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:30.243762 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:30.243845 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:30.244011 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:51:30.244049 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:51:30.245116 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.239317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:34735 (local address 127.24.130.131:35229)
0806 02:51:30.240045 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.240092 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:30.241788 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:30.242456 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:30.242463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:30.242613 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:30.243359 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:51:30.243362 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:30.243419 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:51:30.243518 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:30.243854 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:30.243861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:30.243865 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:51:30.244031 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:51:30.244270 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260806 02:51:30.246341 7275 heartbeater.cc:347] Connected to a master server at 127.24.130.190:34735
I20260806 02:51:30.246559 7275 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:30.247040 7275 heartbeater.cc:511] Master 127.24.130.190:34735 requested a full tablet report, sending...
I20260806 02:51:30.248090 6619 ts_manager.cc:194] Registered new tserver with Master: 37f84f28a39e4c4bac67d0d153d7277a (127.24.130.131:32981)
I20260806 02:51:30.255942 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:51:30.259145 6679 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:30.258125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.24.130.190:34735)
0806 02:51:30.258280 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.258283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.258468 (+ 185us) server_negotiation.cc:407] Connection header received
0806 02:51:30.258578 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.258581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.258616 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.258746 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.258977 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51596: BlockingRecv error: recv got EOF from 127.0.0.1:51596 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:51:30.259384 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 6708
I20260806 02:51:30.281116 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 6901
I20260806 02:51:30.296865 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 7082
I20260806 02:51:30.315089 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 6577
2026-08-06T02:51:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (953 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:30 dist-test-slave-3kk6 krb5kdc[7286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:30 dist-test-slave-3kk6 krb5kdc[7286](info): set up 2 sockets
Aug 06 02:51:30 dist-test-slave-3kk6 krb5kdc[7286](info): commencing operation
krb5kdc: starting...
W20260806 02:51:31.574761 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:51:31 dist-test-slave-3kk6 krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:51:31.714227 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36801
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:31.834698 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:31.835112 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:31.835199 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:31.839466 7308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:31.839600 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:31.839638 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:31.839677 7308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:31.839715 7308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:31.845553 7308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:31.846881 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:31.848017 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:31.856984 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:31.856984 7315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:31.856984 7316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:31.857471 7308 server_base.cc:1034] running on GCE node
I20260806 02:51:31.858148 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:31.859580 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:31.860764 7308 hybrid_clock.cc:648] HybridClock initialized: now 1785984691860743 us; error 41 us; skew 500 ppm
Aug 06 02:51:31 dist-test-slave-3kk6 krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:31.863991 7308 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:51:31.865063 7308 webserver.cc:460] Webserver started at http://127.24.130.190:42983/ using document root <none> and password file <none>
I20260806 02:51:31.865690 7308 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:31.865793 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:31.866061 7308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:31.868435 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "92c64535f70a4bfd908bea979b56950c"
format_stamp: "Formatted at 2026-08-06 02:51:31 on dist-test-slave-3kk6"
server_key: "fb39edbccbb1a8055df2f0028e77938d"
server_key_iv: "3d5478095ccba291326d1da4e04f7789"
server_key_version: "encryptionkey@0"
I20260806 02:51:31.869098 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "92c64535f70a4bfd908bea979b56950c"
format_stamp: "Formatted at 2026-08-06 02:51:31 on dist-test-slave-3kk6"
server_key: "fb39edbccbb1a8055df2f0028e77938d"
server_key_iv: "3d5478095ccba291326d1da4e04f7789"
server_key_version: "encryptionkey@0"
I20260806 02:51:31.874034 7308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 02:51:31.877020 7331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:31.878474 7308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:31.878630 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "92c64535f70a4bfd908bea979b56950c"
format_stamp: "Formatted at 2026-08-06 02:51:31 on dist-test-slave-3kk6"
server_key: "fb39edbccbb1a8055df2f0028e77938d"
server_key_iv: "3d5478095ccba291326d1da4e04f7789"
server_key_version: "encryptionkey@0"
I20260806 02:51:31.878793 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:31.897418 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:31.900796 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:31.901043 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:31.909514 7308 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:36801
I20260806 02:51:31.909538 7409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:36801 every 8 connection(s)
I20260806 02:51:31.910674 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:31.912948 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 7308
I20260806 02:51:31.913065 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:51:31.913389 25098 external_mini_cluster.cc:1428] Setting key d113c796e19b822f77d8da28a45db9a7
I20260806 02:51:31.913749 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:31.921483 7410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c: Bootstrap starting.
Aug 06 02:51:31 dist-test-slave-3kk6 krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:31.924335 7410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:31.925365 7410 log.cc:824] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:31.927942 7410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c: No bootstrap required, opened a new log
I20260806 02:51:31.929507 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.914937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.24.130.190:36801)
0806 02:51:31.915394 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.915415 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.915461 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:51:31.916271 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.916295 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.916731 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.917121 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.917825 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.919358 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.920250 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.920961 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.921123 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.924108 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.924114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.924129 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.924163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.926271 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.926866 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.926871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.926876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.927000 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.927337 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.927339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.927340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.927766 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.927923 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.928306 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.928665 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":118,"threads_started":1}
I20260806 02:51:31.932039 7410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c64535f70a4bfd908bea979b56950c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36801 } }
I20260806 02:51:31.932327 7410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:31.932413 7410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c64535f70a4bfd908bea979b56950c, State: Initialized, Role: FOLLOWER
I20260806 02:51:31.932870 7410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c64535f70a4bfd908bea979b56950c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36801 } }
I20260806 02:51:31.933036 7410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:31.933127 7410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:31.933256 7410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:31.934414 7410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c64535f70a4bfd908bea979b56950c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36801 } }
I20260806 02:51:31.934876 7410 leader_election.cc:302] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c64535f70a4bfd908bea979b56950c; no voters:
I20260806 02:51:31.935379 7410 leader_election.cc:288] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:31.935529 7419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:31.935796 7419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [term 1 LEADER]: Becoming Leader. State: Replica: 92c64535f70a4bfd908bea979b56950c, State: Running, Role: LEADER
I20260806 02:51:31.936159 7419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c64535f70a4bfd908bea979b56950c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36801 } }
I20260806 02:51:31.936944 7410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:31.937954 7420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c64535f70a4bfd908bea979b56950c. Latest consensus state: current_term: 1 leader_uuid: "92c64535f70a4bfd908bea979b56950c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c64535f70a4bfd908bea979b56950c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36801 } } }
I20260806 02:51:31.938109 7420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:31.938573 7421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c64535f70a4bfd908bea979b56950c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c64535f70a4bfd908bea979b56950c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36801 } } }
I20260806 02:51:31.938727 7421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:31.939049 7429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:31.942474 7429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:31.949323 7429 catalog_manager.cc:1347] Generated new cluster ID: 9624da04e97b420fa1bdd108a029051a
I20260806 02:51:31.949462 7429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:31.960124 7429 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:31.961581 7429 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:31.967012 7429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c: Generated new TSK 0
I20260806 02:51:31.967927 7429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:51:32.000253 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36801
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:32.129688 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:32.130010 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:32.130056 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:32.134264 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:32.134343 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:32.134407 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:51:32.140234 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:32.141398 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:32.142436 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:32.153265 7457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:32.153286 7458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:32.153337 7461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:32.153811 7450 server_base.cc:1034] running on GCE node
I20260806 02:51:32.154338 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:32.155154 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:32.156347 7450 hybrid_clock.cc:648] HybridClock initialized: now 1785984692156328 us; error 36 us; skew 500 ppm
Aug 06 02:51:32 dist-test-slave-3kk6 krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984692, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:32.165257 7450 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:51:32.166448 7450 webserver.cc:460] Webserver started at http://127.24.130.129:39773/ using document root <none> and password file <none>
I20260806 02:51:32.167153 7450 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:32.167263 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:32.167526 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:32.169821 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "27392a1040bc471ca0b3addfa34482b3"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "5a39eaa4c767408a46428ab3ffeddd1d"
server_key_iv: "a8e662bdbd4ab93e21be654c35923523"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.170454 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "27392a1040bc471ca0b3addfa34482b3"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "5a39eaa4c767408a46428ab3ffeddd1d"
server_key_iv: "a8e662bdbd4ab93e21be654c35923523"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.174857 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:51:32.177619 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.179013 7450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:32.179189 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "27392a1040bc471ca0b3addfa34482b3"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "5a39eaa4c767408a46428ab3ffeddd1d"
server_key_iv: "a8e662bdbd4ab93e21be654c35923523"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.179333 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:32.196830 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:32.200644 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:32.200908 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:32.201589 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:32.202548 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:32.202623 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.202726 7450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:32.202773 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.213012 7450 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:39757
I20260806 02:51:32.213037 7628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:39757 every 8 connection(s)
I20260806 02:51:32.214046 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:32.214632 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 7450
I20260806 02:51:32.214813 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:32.215165 25098 external_mini_cluster.cc:1428] Setting key 7013c08eed4d6aa06c68a099d5c7f737
Aug 06 02:51:32 dist-test-slave-3kk6 krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984692, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:51:32.230640 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.216365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:49799 (local address 127.24.130.190:36801)
0806 02:51:32.216515 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.216519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.217650 (+ 1131us) server_negotiation.cc:407] Connection header received
0806 02:51:32.219233 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.219236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.219300 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.219425 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:32.220800 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.221367 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.222630 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.222942 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.223068 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.226134 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:32.226137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:32.226139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:32.226166 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:32.227903 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.228555 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.228558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.228560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.228647 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.229051 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.229055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.229058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.229277 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:32.229406 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.230169 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.230365 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:51:32.231503 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.216918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36801 (local address 127.24.130.129:49799)
0806 02:51:32.217493 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.217541 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.218986 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.219638 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.219647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.220088 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:32.220622 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.220637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.221529 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.221531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.222484 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.222495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.223127 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.223131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.223321 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.224072 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:32.224090 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:32.225914 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:32.228063 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:32.228070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:32.228091 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:32.228417 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:32.228750 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:32.228753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:32.228755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:32.228942 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:32.229433 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:32.229445 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:32.229876 (+ 431us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.230078 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.230395 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":160,"threads_started":1}
I20260806 02:51:32.233063 7629 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36801
I20260806 02:51:32.233354 7629 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:32.233933 7629 heartbeater.cc:511] Master 127.24.130.190:36801 requested a full tablet report, sending...
I20260806 02:51:32.235821 7356 ts_manager.cc:194] Registered new tserver with Master: 27392a1040bc471ca0b3addfa34482b3 (127.24.130.129:39757)
I20260806 02:51:32.237099 7356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:49799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:51:32.249363 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36801
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:32.372673 7638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:32.373059 7638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:32.373148 7638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:32.378479 7638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:32.378611 7638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:32.378767 7638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:51:32.384959 7638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:32.386269 7638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:32.387472 7638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:32.398087 7645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:32.398087 7644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:32.398145 7649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:32.398517 7638 server_base.cc:1034] running on GCE node
I20260806 02:51:32.399082 7638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:32.399865 7638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:32.401067 7638 hybrid_clock.cc:648] HybridClock initialized: now 1785984692401047 us; error 37 us; skew 500 ppm
Aug 06 02:51:32 dist-test-slave-3kk6 krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984692, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:32.404354 7638 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:51:32.405323 7638 webserver.cc:460] Webserver started at http://127.24.130.130:33375/ using document root <none> and password file <none>
I20260806 02:51:32.405897 7638 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:32.405961 7638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:32.406143 7638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:32.408458 7638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "9d9ffd56797943598ec5920fefb63e44"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "2c7cf05b133313674cad8cfd643f661e"
server_key_iv: "097d08fed3f85778067ec39fe9032cb0"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.409042 7638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "9d9ffd56797943598ec5920fefb63e44"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "2c7cf05b133313674cad8cfd643f661e"
server_key_iv: "097d08fed3f85778067ec39fe9032cb0"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.413465 7638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:51:32.416456 7661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.417932 7638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:32.418107 7638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "9d9ffd56797943598ec5920fefb63e44"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "2c7cf05b133313674cad8cfd643f661e"
server_key_iv: "097d08fed3f85778067ec39fe9032cb0"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.418251 7638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:32.445335 7638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:32.448933 7638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:32.449163 7638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:32.449832 7638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:32.450806 7638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:32.450866 7638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.450937 7638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:32.450977 7638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.461037 7638 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:36591
I20260806 02:51:32.461062 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:36591 every 8 connection(s)
I20260806 02:51:32.462100 7638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:32.468573 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 7638
I20260806 02:51:32.468683 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:32.469049 25098 external_mini_cluster.cc:1428] Setting key 0656da713919394d6687a6d74e154c34
Aug 06 02:51:32 dist-test-slave-3kk6 krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984692, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:32.479360 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.464265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:45489 (local address 127.24.130.190:36801)
0806 02:51:32.464433 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.464437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.465466 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 02:51:32.467287 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.467290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.467355 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.467474 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:32.468941 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.469546 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.470904 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.471147 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.471274 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.474605 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:32.474608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:32.474610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:32.474639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:32.476424 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.477268 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.477271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.477272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.477368 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.477917 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.477920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.477921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.478140 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:32.478238 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.479068 (+ 830us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.479224 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:51:32.480721 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.464687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36801 (local address 127.24.130.130:45489)
0806 02:51:32.465281 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.465334 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.467013 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.467681 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.467696 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.468131 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:32.468703 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.468725 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.469714 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.469717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.470766 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.470777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.471331 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.471335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.471532 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.472306 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:32.472325 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:32.474321 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:32.476594 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:32.476605 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:32.476631 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:32.477118 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:32.477488 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:32.477494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:32.477498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:32.477747 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:32.478276 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:32.478286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:32.478728 (+ 442us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.478995 (+ 267us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.479443 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:51:32.482210 7819 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36801
I20260806 02:51:32.482554 7819 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:32.483355 7819 heartbeater.cc:511] Master 127.24.130.190:36801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
I20260806 02:51:32.484923 7356 ts_manager.cc:194] Registered new tserver with Master: 9d9ffd56797943598ec5920fefb63e44 (127.24.130.130:36591)
I20260806 02:51:32.485584 7356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:45489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:51:32.502470 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36801
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:32.626405 7829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:32.626828 7829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:32.626912 7829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:32.631137 7829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:32.631218 7829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:32.631282 7829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:51:32.637147 7829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:36801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:51:32.638492 7829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:32.639694 7829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:32.650574 7836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:32.650574 7840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:32.650719 7838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:32.651047 7829 server_base.cc:1034] running on GCE node
I20260806 02:51:32.651527 7829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:32.652307 7829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:32.653512 7829 hybrid_clock.cc:648] HybridClock initialized: now 1785984692653490 us; error 36 us; skew 500 ppm
Aug 06 02:51:32 dist-test-slave-3kk6 krb5kdc[7286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984692, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:32.656908 7829 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:51:32.658016 7829 webserver.cc:460] Webserver started at http://127.24.130.131:37951/ using document root <none> and password file <none>
I20260806 02:51:32.658643 7829 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:32.658803 7829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:32.659072 7829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:32.661358 7829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "d7ad2d0375ed42978785ea5faa8d74e6"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "dd791a930157dbcbe772d9c9a13e632f"
server_key_iv: "70a4534ba6be139dce15e81f17d9893e"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.662016 7829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "d7ad2d0375ed42978785ea5faa8d74e6"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "dd791a930157dbcbe772d9c9a13e632f"
server_key_iv: "70a4534ba6be139dce15e81f17d9893e"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.666373 7829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:51:32.669200 7847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.670552 7829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:32.670755 7829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "d7ad2d0375ed42978785ea5faa8d74e6"
format_stamp: "Formatted at 2026-08-06 02:51:32 on dist-test-slave-3kk6"
server_key: "dd791a930157dbcbe772d9c9a13e632f"
server_key_iv: "70a4534ba6be139dce15e81f17d9893e"
server_key_version: "encryptionkey@0"
I20260806 02:51:32.670902 7829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:32.699326 7829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:32.703377 7829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:32.703624 7829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:32.704314 7829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:32.705225 7829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:32.705298 7829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.705375 7829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:32.705427 7829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:32.715562 7829 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:46677
I20260806 02:51:32.715585 8006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:46677 every 8 connection(s)
I20260806 02:51:32.716572 7829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:32.722018 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 7829
I20260806 02:51:32.722177 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:32.722541 25098 external_mini_cluster.cc:1428] Setting key f75330b92b7df1e1cd58f3e38b144905
Aug 06 02:51:32 dist-test-slave-3kk6 krb5kdc[7286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984692, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:51:32.732604 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.718595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:47491 (local address 127.24.130.190:36801)
0806 02:51:32.718759 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.718762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.719694 (+ 932us) server_negotiation.cc:407] Connection header received
0806 02:51:32.721220 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.721224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.721292 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.721415 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:32.723217 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.723836 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.725172 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.725406 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.725526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.728468 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:32.728472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:32.728474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:32.728502 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:32.730147 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.730905 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.730909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.730910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.731004 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.731393 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.731395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.731396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.731598 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:32.731688 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.732340 (+ 652us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.732471 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:32.733647 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.719047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36801 (local address 127.24.130.131:47491)
0806 02:51:32.719524 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.719572 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.720969 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.721601 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.721608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.722232 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:32.723009 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.723027 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.723979 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.723983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.725040 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.725050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.725571 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.725575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.725768 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.726473 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:32.726490 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:32.728249 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:32.730302 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:32.730309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:32.730326 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:32.730761 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:32.731104 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:32.731107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:32.731108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:32.731291 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:32.731711 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:32.731717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:32.732044 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.732230 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.732685 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260806 02:51:32.735105 8008 heartbeater.cc:347] Connected to a master server at 127.24.130.190:36801
I20260806 02:51:32.735425 8008 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:32.736172 8008 heartbeater.cc:511] Master 127.24.130.190:36801 requested a full tablet report, sending...
I20260806 02:51:32.737460 7356 ts_manager.cc:194] Registered new tserver with Master: d7ad2d0375ed42978785ea5faa8d74e6 (127.24.130.131:46677)
I20260806 02:51:32.738104 7356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:47491
I20260806 02:51:32.749207 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:51:32.760473 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.751639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.24.130.190:36801)
0806 02:51:32.751808 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.751812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.752031 (+ 219us) server_negotiation.cc:407] Connection header received
0806 02:51:32.752120 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.752123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.752173 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.752268 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:32.753044 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.753611 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.754782 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.755041 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.755151 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.756783 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:32.756787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:32.756789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:32.756822 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:32.758480 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.759086 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.759089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.759090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.759188 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:32.759503 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:32.759506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:32.759508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:32.759807 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:32.759932 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.760220 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.760341 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:51:32.764075 7356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:32.767383 7356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:32.784603 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.779000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.24.130.130:36591)
0806 02:51:32.779363 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.779375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.779397 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:51:32.779469 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.779473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.779658 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.780120 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:32.780283 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.781320 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.782880 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.783807 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.783953 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.784183 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.784294 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.784409 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260806 02:51:32.784619 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.777865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:36591 (local address 127.0.0.1:53716)
0806 02:51:32.778788 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.778805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.778912 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.779820 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.779823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.779864 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:32.779959 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.779966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.781455 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.781460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.782723 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.782735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.783964 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.783968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.784088 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.784131 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.784271 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.784420 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"spinlock_wait_cycles":4864,"thread_start_us":113,"threads_started":1}
I20260806 02:51:32.784653 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.778496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:46677 (local address 127.0.0.1:36636)
0806 02:51:32.779737 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.779753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.779857 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.780413 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.780416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.780437 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:32.780539 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.780545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.781699 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.781702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.782936 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.782947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.784219 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.784223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.784329 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.784354 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.784419 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.784490 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":23,"spinlock_wait_cycles":5504,"thread_start_us":116,"threads_started":1}
I20260806 02:51:32.784786 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.777867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.24.130.129:39757)
0806 02:51:32.778229 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.778237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.778867 (+ 630us) server_negotiation.cc:407] Connection header received
0806 02:51:32.778965 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.778971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.779207 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.779378 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:32.779766 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.781013 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.782827 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.783827 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.783977 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.784124 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.784510 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.784610 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
I20260806 02:51:32.785053 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.777667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:39757 (local address 127.0.0.1:37998)
0806 02:51:32.778402 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.778467 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.778624 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.779396 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.779403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.779432 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:32.779603 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.779610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.781152 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.781156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.782612 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.782627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.784187 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.784189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.784287 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.784307 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.784370 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.784419 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":24,"thread_start_us":115,"threads_started":1}
I20260806 02:51:32.785457 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.778630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.24.130.131:46677)
0806 02:51:32.779184 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.779197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.779959 (+ 762us) server_negotiation.cc:407] Connection header received
0806 02:51:32.780049 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.780056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.780239 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.780440 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:32.780640 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.781559 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.783103 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.783933 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.784032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.785149 (+ 1117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.785205 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.785299 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":187,"threads_started":1}
I20260806 02:51:32.788311 7913 tablet_service.cc:1467] Processing CreateTablet for tablet 878128fc2f6e477f99b276ffa1af98b3 (DEFAULT_TABLE table=test-table [id=508e8fba121e4fe99f2f42b94b9e4168]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:32.788327 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 878128fc2f6e477f99b276ffa1af98b3 (DEFAULT_TABLE table=test-table [id=508e8fba121e4fe99f2f42b94b9e4168]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:32.788329 7722 tablet_service.cc:1467] Processing CreateTablet for tablet 878128fc2f6e477f99b276ffa1af98b3 (DEFAULT_TABLE table=test-table [id=508e8fba121e4fe99f2f42b94b9e4168]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:32.789433 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878128fc2f6e477f99b276ffa1af98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:32.789434 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878128fc2f6e477f99b276ffa1af98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:32.789434 7722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878128fc2f6e477f99b276ffa1af98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:32.797747 8030 tablet_bootstrap.cc:492] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Bootstrap starting.
I20260806 02:51:32.799029 8027 tablet_bootstrap.cc:492] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Bootstrap starting.
I20260806 02:51:32.800308 8030 tablet_bootstrap.cc:654] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:32.801138 8030 log.cc:824] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:32.801162 8027 tablet_bootstrap.cc:654] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:32.801967 8028 tablet_bootstrap.cc:492] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: Bootstrap starting.
I20260806 02:51:32.802018 8027 log.cc:824] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:32.803743 8030 tablet_bootstrap.cc:492] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: No bootstrap required, opened a new log
I20260806 02:51:32.803969 8030 ts_tablet_manager.cc:1397] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:51:32.804092 8027 tablet_bootstrap.cc:492] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: No bootstrap required, opened a new log
I20260806 02:51:32.804279 8027 ts_tablet_manager.cc:1397] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:51:32.804301 8028 tablet_bootstrap.cc:654] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:32.805028 8028 log.cc:824] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:32.806845 8028 tablet_bootstrap.cc:492] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: No bootstrap required, opened a new log
I20260806 02:51:32.807047 8028 ts_tablet_manager.cc:1397] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:51:32.808188 8027 raft_consensus.cc:348] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.808188 8030 raft_consensus.cc:348] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.808507 8030 raft_consensus.cc:374] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:32.808507 8027 raft_consensus.cc:374] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:32.808571 8030 raft_consensus.cc:729] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9ffd56797943598ec5920fefb63e44, State: Initialized, Role: FOLLOWER
I20260806 02:51:32.808584 8027 raft_consensus.cc:729] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ad2d0375ed42978785ea5faa8d74e6, State: Initialized, Role: FOLLOWER
I20260806 02:51:32.809089 8027 consensus_queue.cc:260] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.809089 8030 consensus_queue.cc:260] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.810243 8030 ts_tablet_manager.cc:1428] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:51:32.810679 7819 heartbeater.cc:503] Master 127.24.130.190:36801 was elected leader, sending a full tablet report...
I20260806 02:51:32.811229 8028 raft_consensus.cc:348] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.811498 8028 raft_consensus.cc:374] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:32.810458 8027 ts_tablet_manager.cc:1428] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:51:32.811564 8028 raft_consensus.cc:729] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27392a1040bc471ca0b3addfa34482b3, State: Initialized, Role: FOLLOWER
I20260806 02:51:32.811901 8036 raft_consensus.cc:482] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:32.812005 8008 heartbeater.cc:503] Master 127.24.130.190:36801 was elected leader, sending a full tablet report...
I20260806 02:51:32.812040 8036 raft_consensus.cc:504] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.812100 8028 consensus_queue.cc:260] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.813423 8028 ts_tablet_manager.cc:1428] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:51:32.813793 7629 heartbeater.cc:503] Master 127.24.130.190:36801 was elected leader, sending a full tablet report...
I20260806 02:51:32.813961 8036 leader_election.cc:288] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27392a1040bc471ca0b3addfa34482b3 (127.24.130.129:39757), 9d9ffd56797943598ec5920fefb63e44 (127.24.130.130:36591)
I20260806 02:51:32.818444 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.813488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:36591 (local address 127.24.130.131:59969)
0806 02:51:32.814168 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.814186 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.814300 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.814740 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.814743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.814762 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:32.814882 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.814890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.815948 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.815954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.816983 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.816993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.818002 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.818005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.818097 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.818117 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.818218 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.818306 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609}
I20260806 02:51:32.818444 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.813775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:59969 (local address 127.24.130.130:36591)
0806 02:51:32.813921 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.813926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.814396 (+ 470us) server_negotiation.cc:407] Connection header received
0806 02:51:32.814450 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.814454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.814508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.814577 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:32.815015 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.815797 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.817112 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.817729 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.817898 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.817962 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.818235 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.818310 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:32.819208 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878128fc2f6e477f99b276ffa1af98b3" candidate_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9ffd56797943598ec5920fefb63e44" is_pre_election: true
I20260806 02:51:32.819622 7752 raft_consensus.cc:2454] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ad2d0375ed42978785ea5faa8d74e6 in term 0.
I20260806 02:51:32.819800 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.815385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:39757 (local address 127.24.130.131:33025)
0806 02:51:32.815883 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.815899 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:32.815993 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:32.816339 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:32.816342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:32.816359 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:32.816472 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.816479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.817383 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.817387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.818416 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:32.818429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.819364 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.819368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:32.819474 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.819499 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:51:32.819584 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:51:32.819644 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":137,"threads_started":1}
I20260806 02:51:32.819821 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.815796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:33025 (local address 127.24.130.129:39757)
0806 02:51:32.815922 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.815930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.815950 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:51:32.816102 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.816106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.816158 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.816229 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:32.816613 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.817261 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.818562 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.819209 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.819358 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.819413 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.819616 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.819706 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:51:32.820277 7850 leader_election.cc:302] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d9ffd56797943598ec5920fefb63e44, d7ad2d0375ed42978785ea5faa8d74e6; no voters:
I20260806 02:51:32.820362 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878128fc2f6e477f99b276ffa1af98b3" candidate_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27392a1040bc471ca0b3addfa34482b3" is_pre_election: true
I20260806 02:51:32.820591 8036 raft_consensus.cc:2781] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:32.820681 8036 raft_consensus.cc:482] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:32.820715 8036 raft_consensus.cc:3037] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:32.820729 7573 raft_consensus.cc:2454] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ad2d0375ed42978785ea5faa8d74e6 in term 0.
I20260806 02:51:32.821962 8036 raft_consensus.cc:504] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.822392 8036 leader_election.cc:288] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [CANDIDATE]: Term 1 election: Requested vote from peers 27392a1040bc471ca0b3addfa34482b3 (127.24.130.129:39757), 9d9ffd56797943598ec5920fefb63e44 (127.24.130.130:36591)
I20260806 02:51:32.822837 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878128fc2f6e477f99b276ffa1af98b3" candidate_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d9ffd56797943598ec5920fefb63e44"
I20260806 02:51:32.822836 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878128fc2f6e477f99b276ffa1af98b3" candidate_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27392a1040bc471ca0b3addfa34482b3"
I20260806 02:51:32.822962 7752 raft_consensus.cc:3037] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:32.822993 7573 raft_consensus.cc:3037] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:32.824254 7752 raft_consensus.cc:2454] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ad2d0375ed42978785ea5faa8d74e6 in term 1.
I20260806 02:51:32.824256 7573 raft_consensus.cc:2454] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ad2d0375ed42978785ea5faa8d74e6 in term 1.
I20260806 02:51:32.824571 7850 leader_election.cc:302] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d9ffd56797943598ec5920fefb63e44, d7ad2d0375ed42978785ea5faa8d74e6; no voters:
I20260806 02:51:32.824790 8036 raft_consensus.cc:2781] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:32.824922 8036 raft_consensus.cc:686] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 1 LEADER]: Becoming Leader. State: Replica: d7ad2d0375ed42978785ea5faa8d74e6, State: Running, Role: LEADER
I20260806 02:51:32.825268 8036 consensus_queue.cc:237] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } }
I20260806 02:51:32.829015 7356 catalog_manager.cc:5495] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ad2d0375ed42978785ea5faa8d74e6 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ad2d0375ed42978785ea5faa8d74e6" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46677 } health_report { overall_health: HEALTHY } } }
I20260806 02:51:32.852353 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.848429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.24.130.131:46677)
0806 02:51:32.848558 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.848561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.848575 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:51:32.848722 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.848724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.848784 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.848888 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.849166 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.849679 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.850632 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.850930 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.851014 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.851425 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.851909 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.852011 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.852156 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.852220 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:51:32.859179 7573 raft_consensus.cc:1261] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3 [term 1 FOLLOWER]: Refusing update from remote peer d7ad2d0375ed42978785ea5faa8d74e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:32.859179 7752 raft_consensus.cc:1261] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 1 FOLLOWER]: Refusing update from remote peer d7ad2d0375ed42978785ea5faa8d74e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:32.860163 8036 consensus_queue.cc:1035] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27392a1040bc471ca0b3addfa34482b3" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:32.860440 8039 consensus_queue.cc:1035] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d9ffd56797943598ec5920fefb63e44" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:32.868660 8046 mvcc.cc:205] Tried to move back new op lower bound from 7315393301942153216 to 7315393301815275520. Current Snapshot: MvccSnapshot[applied={T|T < 7315393301942153216}]
I20260806 02:51:32.879786 7355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:32.880018 7355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:51:32.883090 7355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92c64535f70a4bfd908bea979b56950c: Sending DeleteTablet for 3 replicas of tablet 878128fc2f6e477f99b276ffa1af98b3
I20260806 02:51:32.884231 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet 878128fc2f6e477f99b276ffa1af98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:37998
I20260806 02:51:32.884291 7913 tablet_service.cc:1514] Processing DeleteTablet for tablet 878128fc2f6e477f99b276ffa1af98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:36636
I20260806 02:51:32.884899 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 7450
I20260806 02:51:32.885071 7722 tablet_service.cc:1514] Processing DeleteTablet for tablet 878128fc2f6e477f99b276ffa1af98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:51:32 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:53716
I20260806 02:51:32.885139 8065 tablet_replica.cc:330] T 878128fc2f6e477f99b276ffa1af98b3 P 27392a1040bc471ca0b3addfa34482b3: stopping tablet replica
I20260806 02:51:32.889178 8066 tablet_replica.cc:330] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: stopping tablet replica
I20260806 02:51:32.889591 8066 raft_consensus.cc:2229] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:51:32.889998 8066 raft_consensus.cc:2258] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:32.892108 8066 ts_tablet_manager.cc:1905] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:32.894801 8066 ts_tablet_manager.cc:1918] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:32.894949 8066 log.cc:1196] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/878128fc2f6e477f99b276ffa1af98b3
I20260806 02:51:32.895337 8066 ts_tablet_manager.cc:1939] T 878128fc2f6e477f99b276ffa1af98b3 P 9d9ffd56797943598ec5920fefb63e44: Deleting consensus metadata
I20260806 02:51:32.896005 8063 tablet_replica.cc:330] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: stopping tablet replica
I20260806 02:51:32.896366 8063 raft_consensus.cc:2229] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 1 LEADER]: Raft consensus shutting down.
W20260806 02:51:32.896859 7337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:39757 (error 108)
I20260806 02:51:32.896907 7335 catalog_manager.cc:4841] TS 9d9ffd56797943598ec5920fefb63e44 (127.24.130.130:36591): tablet 878128fc2f6e477f99b276ffa1af98b3 (table test-table [id=508e8fba121e4fe99f2f42b94b9e4168]) successfully deleted
I20260806 02:51:32.897328 8063 raft_consensus.cc:2258] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:51:32.897864 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 7638
I20260806 02:51:32.898198 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.897779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:39757 (local address 127.0.0.1:38008)
0806 02:51:32.897997 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:32.898078 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 02:51:32.899174 8063 ts_tablet_manager.cc:1905] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:51:32.902029 8063 ts_tablet_manager.cc:1918] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:51:32.902129 8063 log.cc:1196] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/878128fc2f6e477f99b276ffa1af98b3
I20260806 02:51:32.902472 8063 ts_tablet_manager.cc:1939] T 878128fc2f6e477f99b276ffa1af98b3 P d7ad2d0375ed42978785ea5faa8d74e6: Deleting consensus metadata
I20260806 02:51:32.903837 7335 catalog_manager.cc:4841] TS d7ad2d0375ed42978785ea5faa8d74e6 (127.24.130.131:46677): tablet 878128fc2f6e477f99b276ffa1af98b3 (table test-table [id=508e8fba121e4fe99f2f42b94b9e4168]) successfully deleted
W20260806 02:51:32.914518 7337 catalog_manager.cc:4584] TS 27392a1040bc471ca0b3addfa34482b3 (127.24.130.129:39757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 878128fc2f6e477f99b276ffa1af98b3: Network error: Client connection negotiation failed: client connection to 127.24.130.129:39757: connect: Connection refused (error 111)
I20260806 02:51:32.925841 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 7829
I20260806 02:51:32.936328 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 7308
2026-08-06T02:51:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2624 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:33 dist-test-slave-3kk6 krb5kdc[8070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:33 dist-test-slave-3kk6 krb5kdc[8070](info): set up 2 sockets
Aug 06 02:51:33 dist-test-slave-3kk6 krb5kdc[8070](info): commencing operation
krb5kdc: starting...
W20260806 02:51:34.169178 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:51:34 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/rangeradmin_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/rangeradmin_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/rangerlookup_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/rangerlookup_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
I20260806 02:51:34.357374 25098 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGNnxjX/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/postgres -l logfile start
2026-08-06 02:51:36.887 UTC [8111] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:51:36.887 UTC [8111] LOG: listening on IPv4 address "127.24.130.148", port 60281
2026-08-06 02:51:36.891 UTC [8111] LOG: listening on Unix socket "/tmp/.s.PGSQL.60281"
2026-08-06 02:51:36.928 UTC [8116] LOG: database system was shut down at 2026-08-06 02:51:35 UTC
2026-08-06 02:51:36.933 UTC [8111] LOG: database system is ready to accept connections
I20260806 02:51:38.052081 25098 mini_postgres.cc:96] Postgres bound to 60281
2026-08-06 02:51:38.060 UTC [8126] FATAL: database "slave" does not exist
127.24.130.148:60281 - accepting connections
I20260806 02:51:38.061216 25098 mini_ranger.cc:161] Starting Ranger...
I20260806 02:51:38.079042 25098 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:51:38.554628 25098 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:51:38.554764 25098 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-admin
2026-08-06 02:51:38,845 [I] DB FLAVOR :POSTGRES
2026-08-06 02:51:38,846 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:51:38,846 [I] Checking connection..
2026-08-06 02:51:38,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:51:39,105 [I] Checking connection passed.
2026-08-06 02:51:39,106 [I] --------- Verifying version history table ---------
2026-08-06 02:51:39,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:51:39,366 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:51:39,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:51:39,655 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:51:39,655 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:51:39,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:51:39.919 UTC [8217] WARNING: there is no transaction in progress
2026-08-06 02:51:39,926 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:51:39,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:51:40,193 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:51:40,193 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:51:40,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:51:40,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:51:40,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3kk6.c.gcp-upstream.internal','N') ;"
2026-08-06 02:51:40,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:51:41,241 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:51:41,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:51:41,521 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:51:41,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:51:41,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:51:42,035 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:51:42,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:51:43.039 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.057 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.074 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.090 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.110 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.128 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.189 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.197 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.212 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.223 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.234 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.249 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.259 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.266 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.273 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43.280 UTC [8373] WARNING: there is no transaction in progress
2026-08-06 02:51:43,978 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:51:43,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:51:44,227 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:51:44,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:51:44,506 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:51:44,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:51:44,754 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:51:44,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:51:45,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:51:45,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3kk6.c.gcp-upstream.internal';"
2026-08-06 02:51:45,582 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:51:45,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:51:45,845 [I] DB_PATCHES have already been applied
I20260806 02:51:45.851711 25098 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGNnxjX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:51:45.851794 25098 mini_ranger.cc:193] Using host: 127.24.130.148
I20260806 02:51:45.856873 25098 mini_ranger.cc:227] Ranger admin URL: http://127.24.130.148:54481
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:51:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:51:47.236263 25098 mini_ranger.cc:160] Time spent starting Ranger: real 9.175s user 0.000s sys 0.014s
Aug 06, 2026 2:52:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:52:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:52:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:52:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:52:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:52:09.105300 25098 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/rangerkms_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/rangerkms_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.130.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.130.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:52:09 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984729, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:52:09.205888 25098 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:52:09.218299 25098 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:52:10.057789 25098 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:52:10.057911 25098 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms
I20260806 02:52:10.058032 25098 mini_ranger_kms.cc:227] Using postgres at 127.24.130.148:60281
2026-08-06 02:52:10,416 [I] DB FLAVOR :POSTGRES
2026-08-06 02:52:10,416 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:52:10,417 [I] Checking connection
2026-08-06 02:52:10,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:52:10,690 [I] connection success
2026-08-06 02:52:10,690 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:52:10,690 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:52:10,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:52:10,968 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:52:10,968 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:52:10,968 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:52:10,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:60281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:52:11,281 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:52:11.775602 25098 mini_ranger_kms.cc:310] Created kms service
I20260806 02:52:11.986779 25098 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:52:12.034132 25098 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:52:12 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984729, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:52:12.459105 25098 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:52:12.459314 25098 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:52:12.459344 25098 mini_ranger_kms.cc:240] Using host: 127.24.130.148
I20260806 02:52:12.464500 25098 mini_ranger_kms.cc:276] Ranger KMS PID: 8649
I20260806 02:52:12.464640 25098 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.130.148:58539
Aug 06, 2026 2:52:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:52:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:52:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:52:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:52:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:52:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:52:15 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984735, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.130.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:15.246417 25098 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.041s user 0.028s sys 0.313s
I20260806 02:52:15.246577 25098 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:52:15.246757 25098 mini_ranger_kms.cc:401] 127.24.130.148:58539/kms/v1/keys
Aug 06 02:52:15 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984735, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.130.148@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
Aug 06 02:52:15 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984735, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.130.148@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
Aug 06 02:52:18 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:52:18.394126 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43951
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43951
--ranger_config_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:58539/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:18.522073 8724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:18.522364 8724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:18.522405 8724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:18.526436 8724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:18.526510 8724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:18.526530 8724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:18.526546 8724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:18.526562 8724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:18.532245 8724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.130.148:58539/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43951
--ranger_config_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:52:18.533247 8724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:18.534220 8724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:18.547322 8733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:18.547312 8732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:18.547307 8735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:18.547354 8724 server_base.cc:1034] running on GCE node
I20260806 02:52:18.548460 8724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:18.550160 8724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:18.551360 8724 hybrid_clock.cc:648] HybridClock initialized: now 1785984738551351 us; error 46 us; skew 500 ppm
Aug 06 02:52:18 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984738, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:18.559697 8724 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:52:18.560834 8724 webserver.cc:460] Webserver started at http://127.24.130.190:39755/ using document root <none> and password file <none>
I20260806 02:52:18.561424 8724 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:18.561483 8724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:18.561743 8724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:52:18 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984738, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:52:18.611976 8724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "baa1b58a1270445ba2b41a0a1a76c487"
format_stamp: "Formatted at 2026-08-06 02:52:18 on dist-test-slave-3kk6"
server_key: "8dd44e8bb28e724f3d9da9d27e0e2b9e"
server_key_iv: "7bfae386e9c07c009ee6b646e4ac2466"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:18.612668 8724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "baa1b58a1270445ba2b41a0a1a76c487"
format_stamp: "Formatted at 2026-08-06 02:52:18 on dist-test-slave-3kk6"
server_key: "8dd44e8bb28e724f3d9da9d27e0e2b9e"
server_key_iv: "7bfae386e9c07c009ee6b646e4ac2466"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:18.616976 8724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:52:18.635768 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:18.637390 8724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:52:18.637544 8724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "baa1b58a1270445ba2b41a0a1a76c487"
format_stamp: "Formatted at 2026-08-06 02:52:18 on dist-test-slave-3kk6"
server_key: "8dd44e8bb28e724f3d9da9d27e0e2b9e"
server_key_iv: "7bfae386e9c07c009ee6b646e4ac2466"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:18.637694 8724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:18.651932 8724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:18.655303 8724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:18.655535 8724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:18.665480 8724 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43951
I20260806 02:52:18.665505 8822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43951 every 8 connection(s)
I20260806 02:52:18.666548 8724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:18.669836 8823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:18.674158 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 8724
I20260806 02:52:18.674314 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:52:18.674615 25098 external_mini_cluster.cc:1428] Setting key a7fe64a198a4586517b783f8542401b4
I20260806 02:52:18.675954 8823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487: Bootstrap starting.
I20260806 02:52:18.678328 8823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:18.679174 8823 log.cc:824] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:18.681156 8823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487: No bootstrap required, opened a new log
Aug 06 02:52:18 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:52:18.685321 8823 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa1b58a1270445ba2b41a0a1a76c487" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43951 } }
I20260806 02:52:18.685598 8823 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:18.685654 8823 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa1b58a1270445ba2b41a0a1a76c487, State: Initialized, Role: FOLLOWER
I20260806 02:52:18.686124 8823 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa1b58a1270445ba2b41a0a1a76c487" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43951 } }
I20260806 02:52:18.686260 8823 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:18.686359 8823 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:18.686487 8823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:18.687638 8823 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa1b58a1270445ba2b41a0a1a76c487" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43951 } }
I20260806 02:52:18.687995 8823 leader_election.cc:302] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa1b58a1270445ba2b41a0a1a76c487; no voters:
I20260806 02:52:18.688367 8823 leader_election.cc:288] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:18.688515 8829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:18.688761 8829 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [term 1 LEADER]: Becoming Leader. State: Replica: baa1b58a1270445ba2b41a0a1a76c487, State: Running, Role: LEADER
I20260806 02:52:18.689177 8829 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa1b58a1270445ba2b41a0a1a76c487" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43951 } }
I20260806 02:52:18.689687 8823 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:18.691247 8832 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa1b58a1270445ba2b41a0a1a76c487. Latest consensus state: current_term: 1 leader_uuid: "baa1b58a1270445ba2b41a0a1a76c487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa1b58a1270445ba2b41a0a1a76c487" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43951 } } }
I20260806 02:52:18.691298 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:18.676208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.24.130.190:43951)
0806 02:52:18.676686 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:18.676702 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:18.676747 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:52:18.677508 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:18.677527 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:18.677882 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:18.678239 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:18.679215 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:18.680265 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:18.681193 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:18.681742 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:18.681927 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:18.684658 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:18.684663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:18.684678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:18.684707 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:18.687082 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:18.687553 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:18.687559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:18.687565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:18.687688 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:18.688048 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:18.688052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:18.688054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:18.688719 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:18.689701 (+ 982us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:18.689869 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:52:18.690156 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":155,"threads_started":1}
I20260806 02:52:18.691286 8830 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa1b58a1270445ba2b41a0a1a76c487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa1b58a1270445ba2b41a0a1a76c487" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43951 } } }
I20260806 02:52:18.691375 8832 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:18.691406 8830 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:18.692397 8841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:18.693008 8823 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:52:18.695825 8841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:18.706900 8841 catalog_manager.cc:1347] Generated new cluster ID: 133aab5ffd004b7c8b67ccbf42eb1a10
I20260806 02:52:18.707001 8841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:18.721024 8841 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:18.722390 8841 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:18.764892 8841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487: Generated new TSK 0
I20260806 02:52:18.765861 8841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:52:19.616132 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:58539/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43951
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:19.805055 8882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:19.805420 8882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:19.805523 8882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:19.809937 8882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:19.810046 8882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:19.810155 8882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:52:19.817386 8882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.130.148:58539/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:52:19.819046 8882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:19.820674 8882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:19.834530 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:19.834671 8897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:19.834530 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:19.835321 8882 server_base.cc:1034] running on GCE node
I20260806 02:52:19.835767 8882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:19.836542 8882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:19.837726 8882 hybrid_clock.cc:648] HybridClock initialized: now 1785984739837706 us; error 40 us; skew 500 ppm
Aug 06 02:52:19 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984739, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:19.840984 8882 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:52:19.842149 8882 webserver.cc:460] Webserver started at http://127.24.130.129:36333/ using document root <none> and password file <none>
I20260806 02:52:19.842906 8882 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:19.843006 8882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:19.843271 8882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:52:19 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984739, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:52:19.859413 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "f876edaea6bc4a0980eda72f1410611c"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-3kk6"
server_key: "bf81cb9f4d387b7f0f19c948e8c11311"
server_key_iv: "586c5f2c9783fc3b71e2c7a60c064288"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:19.860204 8882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "f876edaea6bc4a0980eda72f1410611c"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-3kk6"
server_key: "bf81cb9f4d387b7f0f19c948e8c11311"
server_key_iv: "586c5f2c9783fc3b71e2c7a60c064288"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:19.868350 8882 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260806 02:52:19.884852 8905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.887029 8882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:52:19.887197 8882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "f876edaea6bc4a0980eda72f1410611c"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-3kk6"
server_key: "bf81cb9f4d387b7f0f19c948e8c11311"
server_key_iv: "586c5f2c9783fc3b71e2c7a60c064288"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:19.887388 8882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:19.917338 8882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:19.921542 8882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:19.921833 8882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:19.922804 8882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:19.924069 8882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:19.924247 8882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.924400 8882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:19.924496 8882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.942598 8882 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:39807
I20260806 02:52:19.942629 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:39807 every 8 connection(s)
I20260806 02:52:19.943742 8882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:19.950474 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 8882
I20260806 02:52:19.950645 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:19.951038 25098 external_mini_cluster.cc:1428] Setting key 95abe1b5671251552533e362c2eb393b
Aug 06 02:52:19 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984739, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:52:19.981580 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:19.947025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43951 (local address 127.24.130.129:34673)
0806 02:52:19.947702 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:19.947795 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:19.952767 (+ 4972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:19.958455 (+ 5688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:19.958470 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:19.959065 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:19.960016 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:19.960043 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.961473 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.961477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:19.963133 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:19.963148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.963946 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.963950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:19.964196 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:19.965058 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:19.965080 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:19.967172 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:19.976378 (+ 9206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:19.976388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:19.976414 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:19.978255 (+ 1841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:19.978816 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:19.978819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:19.978822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:19.979080 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:19.979668 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:19.979678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:19.980150 (+ 472us) client_negotiation.cc:769] Sending connection context
0806 02:52:19.980382 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 02:52:19.980695 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":165,"threads_started":1}
I20260806 02:52:19.982863 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:19.946403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:34673 (local address 127.24.130.190:43951)
0806 02:52:19.957800 (+ 11397us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:19.957805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:19.957829 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:52:19.957902 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:19.957906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:19.957968 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:19.960254 (+ 2286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:19.960403 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.961285 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.963355 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.963739 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.964681 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:19.967428 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:19.967432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:19.967435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:19.967467 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:19.976119 (+ 8652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:19.978510 (+ 2391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:19.978514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:19.978516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:19.978673 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:19.979222 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:19.979226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:19.979227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:19.979525 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:19.982384 (+ 2859us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:19.982496 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 02:52:19.982686 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11237,"thread_start_us":118,"threads_started":1}
I20260806 02:52:19.984334 9081 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43951
I20260806 02:52:19.984651 9081 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:19.985298 9081 heartbeater.cc:511] Master 127.24.130.190:43951 requested a full tablet report, sending...
I20260806 02:52:19.987352 8775 ts_manager.cc:194] Registered new tserver with Master: f876edaea6bc4a0980eda72f1410611c (127.24.130.129:39807)
I20260806 02:52:19.988909 8775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:34673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:52:20.014602 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:58539/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43951
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:52:20.181962 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:20.182363 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:20.182440 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:20.188586 9090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:20.188689 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:20.188783 9090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:52:20.196720 9090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.130.148:58539/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:52:20.198359 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:20.199788 9090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:20.212940 9103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:20.214392 9090 server_base.cc:1034] running on GCE node
W20260806 02:52:20.214735 9101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.214874 9106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:20.215479 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:20.216367 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:20.217576 9090 hybrid_clock.cc:648] HybridClock initialized: now 1785984740217554 us; error 38 us; skew 500 ppm
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:20.221735 9090 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:52:20.223024 9090 webserver.cc:460] Webserver started at http://127.24.130.130:33509/ using document root <none> and password file <none>
I20260806 02:52:20.223821 9090 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:20.223959 9090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:20.224257 9090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:52:20.245877 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "244e672ada294331ac50725266782dc9"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-3kk6"
server_key: "9a436a23df6250c172e4369f037b1bc9"
server_key_iv: "f00ee5b5ccaf2c5de221597dacdeba0f"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:20.246994 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "244e672ada294331ac50725266782dc9"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-3kk6"
server_key: "9a436a23df6250c172e4369f037b1bc9"
server_key_iv: "f00ee5b5ccaf2c5de221597dacdeba0f"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:20.253111 9090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260806 02:52:20.275806 9116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.277648 9090 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 02:52:20.277822 9090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "244e672ada294331ac50725266782dc9"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-3kk6"
server_key: "9a436a23df6250c172e4369f037b1bc9"
server_key_iv: "f00ee5b5ccaf2c5de221597dacdeba0f"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:20.277984 9090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:20.300678 9090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:20.305037 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:20.305336 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:20.306442 9090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:20.307749 9090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:20.307823 9090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.307886 9090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:20.307956 9090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 02:52:20.321419 9090 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:45951
I20260806 02:52:20.322403 9291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:45951 every 8 connection(s)
I20260806 02:52:20.322891 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:20.327338 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 9090
I20260806 02:52:20.327481 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:20.327855 25098 external_mini_cluster.cc:1428] Setting key b0694009f5487aeb58ce1cb5295131e3
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
I20260806 02:52:20.349671 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.327115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:44119 (local address 127.24.130.190:43951)
0806 02:52:20.327249 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.327254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.327287 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:52:20.328068 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.328072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.328129 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.328185 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.330021 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.330932 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.336880 (+ 5948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.337240 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.337340 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.341335 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.341340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.341343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.341378 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.343796 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.344512 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.344516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.344518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.344610 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.345189 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.345192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.345194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.345486 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.345574 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.349272 (+ 3698us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.349462 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:52:20.354278 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.325867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43951 (local address 127.24.130.130:44119)
0806 02:52:20.326523 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.326587 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.327790 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.328614 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.328623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.329136 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:20.329843 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.329863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.334888 (+ 5025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.334892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.336675 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.336690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.337593 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.337597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.337836 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.339130 (+ 1294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:20.339159 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:20.341094 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:20.343947 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.343955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.343974 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.344371 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.344859 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.344862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.344864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.345065 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.348534 (+ 3469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:20.348542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:20.348939 (+ 397us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.350105 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.350515 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":160,"threads_started":1}
I20260806 02:52:20.355878 9293 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43951
I20260806 02:52:20.356164 9293 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:20.356796 9293 heartbeater.cc:511] Master 127.24.130.190:43951 requested a full tablet report, sending...
I20260806 02:52:20.358105 8775 ts_manager.cc:194] Registered new tserver with Master: 244e672ada294331ac50725266782dc9 (127.24.130.130:45951)
I20260806 02:52:20.358824 8775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:44119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:52:20.366106 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:58539/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43951
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:20.539753 9299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:20.540246 9299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:20.540369 9299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
W20260806 02:52:20.546612 9299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:20.546826 9299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:20.546962 9299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:52:20.555613 9299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.130.148:58539/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:52:20.557325 9299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:20.559000 9299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:20.577216 9309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.577845 9307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.578132 9306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:20.579635 9299 server_base.cc:1034] running on GCE node
I20260806 02:52:20.580202 9299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:20.581040 9299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:20.582278 9299 hybrid_clock.cc:648] HybridClock initialized: now 1785984740582210 us; error 85 us; skew 500 ppm
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:20.586515 9299 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:52:20.587791 9299 webserver.cc:460] Webserver started at http://127.24.130.131:33679/ using document root <none> and password file <none>
I20260806 02:52:20.588603 9299 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:20.588758 9299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:20.589077 9299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:52:20.620476 9299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "ba15b84eb54f49199614ece198ba25d4"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-3kk6"
server_key: "958d8efd2c5419e9ec39359edbdeeb64"
server_key_iv: "6ef250b48b4f210717de0d60812d29a4"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:20.623725 9299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "ba15b84eb54f49199614ece198ba25d4"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-3kk6"
server_key: "958d8efd2c5419e9ec39359edbdeeb64"
server_key_iv: "6ef250b48b4f210717de0d60812d29a4"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:20.629866 9299 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 02:52:20.675189 9317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.677121 9299 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260806 02:52:20.677287 9299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "ba15b84eb54f49199614ece198ba25d4"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-3kk6"
server_key: "958d8efd2c5419e9ec39359edbdeeb64"
server_key_iv: "6ef250b48b4f210717de0d60812d29a4"
server_key_version: "kuduclusterkey@0"
I20260806 02:52:20.677452 9299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:20.699371 9299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:20.703428 9299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:20.703758 9299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:20.704514 9299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:20.705673 9299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:20.705794 9299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.705904 9299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:20.705971 9299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.718376 9299 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:36937
I20260806 02:52:20.719718 9299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:20.725826 9475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:36937 every 8 connection(s)
I20260806 02:52:20.728639 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 9299
I20260806 02:52:20.728794 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:20.729139 25098 external_mini_cluster.cc:1428] Setting key bfa7a4d7067e33c3c6131fb4f1f4c14e
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:52:20.747592 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.732122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:37551 (local address 127.24.130.190:43951)
0806 02:52:20.732253 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.732257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.733140 (+ 883us) server_negotiation.cc:407] Connection header received
0806 02:52:20.734461 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.734465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.734518 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.734568 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.736500 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.737365 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.739182 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.739545 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.739628 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.743130 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.743134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.743136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.743166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.745005 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.745742 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.745746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.745748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.745838 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.746253 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.746256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.746258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.746483 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.746561 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.747293 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.747425 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:52:20.749534 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.730819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43951 (local address 127.24.130.131:37551)
0806 02:52:20.732975 (+ 2156us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.733040 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.734205 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.735044 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.735053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.735538 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:20.736318 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.736338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.737531 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.737534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.739030 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.739041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.739864 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.739868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.740076 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.740898 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:20.740921 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:20.742922 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:20.745152 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.745159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.745178 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.745604 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.745951 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.745954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.745956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.746143 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.746628 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:20.746635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:20.747005 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.748076 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.748437 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"thread_start_us":222,"threads_started":1}
I20260806 02:52:20.751247 9476 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43951
I20260806 02:52:20.751492 9476 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:20.752094 9476 heartbeater.cc:511] Master 127.24.130.190:43951 requested a full tablet report, sending...
I20260806 02:52:20.753283 8775 ts_manager.cc:194] Registered new tserver with Master: ba15b84eb54f49199614ece198ba25d4 (127.24.130.131:36937)
I20260806 02:52:20.754031 8775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:37551
I20260806 02:52:20.755489 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:52:20.769858 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.758071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.24.130.190:43951)
0806 02:52:20.758241 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.758246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.759129 (+ 883us) server_negotiation.cc:407] Connection header received
0806 02:52:20.759369 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.759373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.759426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.759486 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.760818 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.761645 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.762951 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.763298 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.763388 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.765629 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.765633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.765636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.765667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.767666 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.768263 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.768267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.768269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.768367 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.768708 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.768712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.768713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.768963 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.769068 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.769509 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.769650 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:52:20.775617 8774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:20.781339 8774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:20.801976 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.794193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:45951 (local address 127.0.0.1:54924)
0806 02:52:20.794766 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.794822 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.794955 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.796718 (+ 1763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.796724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.796749 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:20.796882 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.796888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.798866 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.798873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.800307 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.800318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.801463 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.801467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.801574 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.801620 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.801715 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.801827 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260806 02:52:20.802605 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.795618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.24.130.130:45951)
0806 02:52:20.795916 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.795933 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.795959 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:52:20.796046 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.796051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.796271 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.796519 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:20.797019 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.798555 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.800491 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.801325 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.802194 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.802332 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.802383 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.802473 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260806 02:52:20.806087 9194 tablet_service.cc:1467] Processing CreateTablet for tablet fafb920d31884984a1015d13aaa023cc (DEFAULT_TABLE table=test-table [id=7da8c004a53e4bb98202a5e7a5bb4a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:20.806905 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.796110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:39807 (local address 127.0.0.1:48380)
0806 02:52:20.797242 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.797255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.797325 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.798217 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.798220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.798238 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:20.798324 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.798330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.800733 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.800736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.804397 (+ 3661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.804408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.806411 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.806415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.806515 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.806544 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.806600 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.806705 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":125,"threads_started":1}
I20260806 02:52:20.807596 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb920d31884984a1015d13aaa023cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:20.808868 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.796267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.24.130.129:39807)
0806 02:52:20.797544 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.797551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.797593 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:52:20.797680 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.797685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.797896 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.798041 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:20.798468 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.800568 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.804569 (+ 4001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.805507 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.807446 (+ 1939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.808513 (+ 1067us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.808566 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.808665 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":243,"threads_started":1}
I20260806 02:52:20.814224 8979 tablet_service.cc:1467] Processing CreateTablet for tablet fafb920d31884984a1015d13aaa023cc (DEFAULT_TABLE table=test-table [id=7da8c004a53e4bb98202a5e7a5bb4a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:20.816331 8979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb920d31884984a1015d13aaa023cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:20.824086 9505 tablet_bootstrap.cc:492] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: Bootstrap starting.
I20260806 02:52:20.825840 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.795189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:36937 (local address 127.0.0.1:54086)
0806 02:52:20.796944 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.796959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.797053 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.809168 (+ 12115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.809183 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.809207 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:20.809312 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.809319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.822792 (+ 13473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.822796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.824056 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.824071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.825397 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.825401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.825505 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.825537 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.825596 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.825685 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1688,"thread_start_us":92,"threads_started":1}
I20260806 02:52:20.826692 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.795348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.24.130.131:36937)
0806 02:52:20.805861 (+ 10513us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.805888 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.805915 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:52:20.806000 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.806005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.806220 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.809508 (+ 3288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:20.809683 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.810889 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.824318 (+ 13429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.825203 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.826190 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.826357 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.826417 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.826518 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10390,"thread_start_us":117,"threads_started":1}
I20260806 02:52:20.827314 9505 tablet_bootstrap.cc:654] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:20.828466 9505 log.cc:824] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:20.829051 9501 tablet_bootstrap.cc:492] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Bootstrap starting.
I20260806 02:52:20.830078 9380 tablet_service.cc:1467] Processing CreateTablet for tablet fafb920d31884984a1015d13aaa023cc (DEFAULT_TABLE table=test-table [id=7da8c004a53e4bb98202a5e7a5bb4a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:20.831314 9380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafb920d31884984a1015d13aaa023cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:20.833616 9501 tablet_bootstrap.cc:654] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:20.834625 9501 log.cc:824] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:20.837276 9505 tablet_bootstrap.cc:492] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: No bootstrap required, opened a new log
I20260806 02:52:20.837566 9505 ts_tablet_manager.cc:1397] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 02:52:20.840898 9513 tablet_bootstrap.cc:492] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Bootstrap starting.
I20260806 02:52:20.844708 9513 tablet_bootstrap.cc:654] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:20.845556 9505 raft_consensus.cc:348] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.845966 9513 log.cc:824] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:20.846088 9505 raft_consensus.cc:374] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:20.846218 9505 raft_consensus.cc:729] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f876edaea6bc4a0980eda72f1410611c, State: Initialized, Role: FOLLOWER
I20260806 02:52:20.846958 9505 consensus_queue.cc:260] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.848342 9505 ts_tablet_manager.cc:1428] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260806 02:52:20.848763 9513 tablet_bootstrap.cc:492] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: No bootstrap required, opened a new log
I20260806 02:52:20.848913 9081 heartbeater.cc:503] Master 127.24.130.190:43951 was elected leader, sending a full tablet report...
I20260806 02:52:20.848979 9513 ts_tablet_manager.cc:1397] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260806 02:52:20.852140 9501 tablet_bootstrap.cc:492] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: No bootstrap required, opened a new log
I20260806 02:52:20.852455 9501 ts_tablet_manager.cc:1397] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.004s
I20260806 02:52:20.853734 9513 raft_consensus.cc:348] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.854048 9513 raft_consensus.cc:374] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:20.854115 9513 raft_consensus.cc:729] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba15b84eb54f49199614ece198ba25d4, State: Initialized, Role: FOLLOWER
I20260806 02:52:20.854691 9513 consensus_queue.cc:260] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.855818 9513 ts_tablet_manager.cc:1428] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:52:20.856191 9476 heartbeater.cc:503] Master 127.24.130.190:43951 was elected leader, sending a full tablet report...
I20260806 02:52:20.857276 9501 raft_consensus.cc:348] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.857642 9501 raft_consensus.cc:374] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:20.857760 9501 raft_consensus.cc:729] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244e672ada294331ac50725266782dc9, State: Initialized, Role: FOLLOWER
I20260806 02:52:20.858350 9501 consensus_queue.cc:260] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.859830 9501 ts_tablet_manager.cc:1428] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 02:52:20.860244 9293 heartbeater.cc:503] Master 127.24.130.190:43951 was elected leader, sending a full tablet report...
I20260806 02:52:20.891165 9515 raft_consensus.cc:482] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:20.891505 9515 raft_consensus.cc:504] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.893184 9515 leader_election.cc:288] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244e672ada294331ac50725266782dc9 (127.24.130.130:45951), ba15b84eb54f49199614ece198ba25d4 (127.24.130.131:36937)
I20260806 02:52:20.901181 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.893557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:36937 (local address 127.24.130.129:35511)
0806 02:52:20.894632 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.894667 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.895008 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.895380 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.895383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.895398 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:20.895573 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.895580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.897980 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.897986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.899539 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.899560 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.900660 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.900664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.900777 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.900804 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.900886 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.900957 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":48,"thread_start_us":217,"threads_started":1}
I20260806 02:52:20.901487 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.893921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:34517 (local address 127.24.130.130:45951)
0806 02:52:20.894049 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.894053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.894455 (+ 402us) server_negotiation.cc:407] Connection header received
0806 02:52:20.895737 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.895741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.895799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.895862 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:20.896265 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.897115 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.900156 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.900766 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.900869 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.900920 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.901286 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.901364 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:52:20.901963 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.894088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:35511 (local address 127.24.130.131:36937)
0806 02:52:20.894230 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.894235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.894779 (+ 544us) server_negotiation.cc:407] Connection header received
0806 02:52:20.895132 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.895136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.895185 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.895235 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:20.897020 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.897777 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.899719 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.900328 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.900465 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.900523 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.901756 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.901841 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:52:20.902094 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.893447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:45951 (local address 127.24.130.129:34517)
0806 02:52:20.894347 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.894363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.894893 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.896000 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.896002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.896020 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:20.896130 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.896136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.897244 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.897248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.899952 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.899961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.901019 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.901022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.901142 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.901164 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.901833 (+ 669us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.901909 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260806 02:52:20.902642 9404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb920d31884984a1015d13aaa023cc" candidate_uuid: "f876edaea6bc4a0980eda72f1410611c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba15b84eb54f49199614ece198ba25d4" is_pre_election: true
I20260806 02:52:20.902714 9223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb920d31884984a1015d13aaa023cc" candidate_uuid: "f876edaea6bc4a0980eda72f1410611c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244e672ada294331ac50725266782dc9" is_pre_election: true
I20260806 02:52:20.903029 9223 raft_consensus.cc:2454] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f876edaea6bc4a0980eda72f1410611c in term 0.
I20260806 02:52:20.903040 9404 raft_consensus.cc:2454] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f876edaea6bc4a0980eda72f1410611c in term 0.
I20260806 02:52:20.903543 8907 leader_election.cc:302] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244e672ada294331ac50725266782dc9, f876edaea6bc4a0980eda72f1410611c; no voters:
I20260806 02:52:20.903887 9515 raft_consensus.cc:2781] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:20.903960 9515 raft_consensus.cc:482] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:20.903996 9515 raft_consensus.cc:3037] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:20.905474 9515 raft_consensus.cc:504] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.905840 9515 leader_election.cc:288] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [CANDIDATE]: Term 1 election: Requested vote from peers 244e672ada294331ac50725266782dc9 (127.24.130.130:45951), ba15b84eb54f49199614ece198ba25d4 (127.24.130.131:36937)
I20260806 02:52:20.906642 9223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb920d31884984a1015d13aaa023cc" candidate_uuid: "f876edaea6bc4a0980eda72f1410611c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244e672ada294331ac50725266782dc9"
I20260806 02:52:20.906811 9223 raft_consensus.cc:3037] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:20.908180 9223 raft_consensus.cc:2454] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f876edaea6bc4a0980eda72f1410611c in term 1.
I20260806 02:52:20.908491 9404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafb920d31884984a1015d13aaa023cc" candidate_uuid: "f876edaea6bc4a0980eda72f1410611c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba15b84eb54f49199614ece198ba25d4"
I20260806 02:52:20.908581 8907 leader_election.cc:302] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244e672ada294331ac50725266782dc9, f876edaea6bc4a0980eda72f1410611c; no voters:
I20260806 02:52:20.908661 9404 raft_consensus.cc:3037] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:20.908785 9515 raft_consensus.cc:2781] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:20.908988 9515 raft_consensus.cc:686] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [term 1 LEADER]: Becoming Leader. State: Replica: f876edaea6bc4a0980eda72f1410611c, State: Running, Role: LEADER
I20260806 02:52:20.909354 9515 consensus_queue.cc:237] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } }
I20260806 02:52:20.910125 9404 raft_consensus.cc:2454] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f876edaea6bc4a0980eda72f1410611c in term 1.
I20260806 02:52:20.917766 8774 catalog_manager.cc:5495] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c reported cstate change: term changed from 0 to 1, leader changed from <none> to f876edaea6bc4a0980eda72f1410611c (127.24.130.129). New cstate: current_term: 1 leader_uuid: "f876edaea6bc4a0980eda72f1410611c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f876edaea6bc4a0980eda72f1410611c" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 39807 } health_report { overall_health: HEALTHY } } }
Aug 06 02:52:20 dist-test-slave-3kk6 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:52:20.933530 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.928415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.24.130.129:39807)
0806 02:52:20.928620 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.928625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.928642 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:52:20.928784 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.928787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.928854 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.928969 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:20.929396 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.930122 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.931474 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.931820 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.931932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.932451 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:20.933044 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:20.933202 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.933308 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.933377 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 02:52:20.941871 9404 raft_consensus.cc:1261] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 1 FOLLOWER]: Refusing update from remote peer f876edaea6bc4a0980eda72f1410611c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:20.941871 9223 raft_consensus.cc:1261] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 1 FOLLOWER]: Refusing update from remote peer f876edaea6bc4a0980eda72f1410611c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:20.942797 9526 consensus_queue.cc:1035] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [LEADER]: Connected to new peer: Peer: permanent_uuid: "244e672ada294331ac50725266782dc9" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:20.942987 9515 consensus_queue.cc:1035] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba15b84eb54f49199614ece198ba25d4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:52:20.946902 9082 tablet.cc:2369] T fafb920d31884984a1015d13aaa023cc P f876edaea6bc4a0980eda72f1410611c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:20.954988 9530 mvcc.cc:205] Tried to move back new op lower bound from 7315393498887561216 to 7315393498768007168. Current Snapshot: MvccSnapshot[applied={T|T < 7315393498887561216}]
I20260806 02:52:20.967150 9532 mvcc.cc:205] Tried to move back new op lower bound from 7315393498887561216 to 7315393498768007168. Current Snapshot: MvccSnapshot[applied={T|T < 7315393498887561216}]
I20260806 02:52:20.971827 8774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:20.972039 8774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:20.974862 8774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baa1b58a1270445ba2b41a0a1a76c487: Sending DeleteTablet for 3 replicas of tablet fafb920d31884984a1015d13aaa023cc
I20260806 02:52:20.976022 9194 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb920d31884984a1015d13aaa023cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:20 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:54924
I20260806 02:52:20.976609 8979 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb920d31884984a1015d13aaa023cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:20 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48380
I20260806 02:52:20.976647 9380 tablet_service.cc:1514] Processing DeleteTablet for tablet fafb920d31884984a1015d13aaa023cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:20 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:54086
I20260806 02:52:20.977816 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 8882
I20260806 02:52:20.992103 9545 tablet_replica.cc:330] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: stopping tablet replica
I20260806 02:52:20.992666 9545 raft_consensus.cc:2229] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:20.993116 9545 raft_consensus.cc:2258] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:20.993134 9547 tablet_replica.cc:330] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: stopping tablet replica
I20260806 02:52:21.000536 9545 ts_tablet_manager.cc:1905] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:21.001049 9547 raft_consensus.cc:2229] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:21.001446 9547 raft_consensus.cc:2258] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:21.003142 9547 ts_tablet_manager.cc:1905] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:21.003645 9545 ts_tablet_manager.cc:1918] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:21.003767 9545 log.cc:1196] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/fafb920d31884984a1015d13aaa023cc
I20260806 02:52:21.004141 9545 ts_tablet_manager.cc:1939] T fafb920d31884984a1015d13aaa023cc P 244e672ada294331ac50725266782dc9: Deleting consensus metadata
I20260806 02:52:21.005576 8750 catalog_manager.cc:4841] TS 244e672ada294331ac50725266782dc9 (127.24.130.130:45951): tablet fafb920d31884984a1015d13aaa023cc (table test-table [id=7da8c004a53e4bb98202a5e7a5bb4a6a]) successfully deleted
W20260806 02:52:21.006516 8754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:39807 (error 108)
I20260806 02:52:21.006531 9547 ts_tablet_manager.cc:1918] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:21.007014 9547 log.cc:1196] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/fafb920d31884984a1015d13aaa023cc
I20260806 02:52:21.007557 9547 ts_tablet_manager.cc:1939] T fafb920d31884984a1015d13aaa023cc P ba15b84eb54f49199614ece198ba25d4: Deleting consensus metadata
I20260806 02:52:21.007663 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 9090
I20260806 02:52:21.014508 8749 catalog_manager.cc:4841] TS ba15b84eb54f49199614ece198ba25d4 (127.24.130.131:36937): tablet fafb920d31884984a1015d13aaa023cc (table test-table [id=7da8c004a53e4bb98202a5e7a5bb4a6a]) successfully deleted
I20260806 02:52:21.021390 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:21.020960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:39807 (local address 127.0.0.1:48404)
0806 02:52:21.021193 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:21.021288 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:39807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260806 02:52:21.021613 8754 catalog_manager.cc:4584] TS f876edaea6bc4a0980eda72f1410611c (127.24.130.129:39807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fafb920d31884984a1015d13aaa023cc: Network error: Client connection negotiation failed: client connection to 127.24.130.129:39807: connect: Connection refused (error 111)
I20260806 02:52:21.023648 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 9299
I20260806 02:52:21.039038 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 8724
I20260806 02:52:21.054917 25098 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:52:21.401213 25098 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:52:21.401351 25098 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:52:21.757669 25098 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:52:21.758 UTC [8111] LOG: received smart shutdown request
2026-08-06 02:52:21.762 UTC [8111] LOG: background worker "logical replication launcher" (PID 8124) exited with exit code 1
2026-08-06 02:52:21.762 UTC [8117] LOG: shutting down
2026-08-06 02:52:21.936 UTC [8111] LOG: database system is shut down
2026-08-06T02:52:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49120 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:22 dist-test-slave-3kk6 krb5kdc[9565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:22 dist-test-slave-3kk6 krb5kdc[9565](info): set up 2 sockets
Aug 06 02:52:22 dist-test-slave-3kk6 krb5kdc[9565](info): commencing operation
krb5kdc: starting...
W20260806 02:52:23.286510 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:52:23 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/rangeradmin_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/rangeradmin_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/rangerlookup_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/rangerlookup_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
I20260806 02:52:23.474043 25098 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGNnxjX/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/postgres -l logfile start
2026-08-06 02:52:26.026 UTC [9614] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:52:26.026 UTC [9614] LOG: listening on IPv4 address "127.24.130.148", port 35947
2026-08-06 02:52:26.030 UTC [9614] LOG: listening on Unix socket "/tmp/.s.PGSQL.35947"
2026-08-06 02:52:26.052 UTC [9618] LOG: database system was shut down at 2026-08-06 02:52:24 UTC
2026-08-06 02:52:26.056 UTC [9614] LOG: database system is ready to accept connections
I20260806 02:52:27.233140 25098 mini_postgres.cc:96] Postgres bound to 35947
2026-08-06 02:52:27.241 UTC [9627] FATAL: database "slave" does not exist
127.24.130.148:35947 - accepting connections
I20260806 02:52:27.241699 25098 mini_ranger.cc:161] Starting Ranger...
I20260806 02:52:27.259019 25098 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:52:27.750855 25098 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:52:27.750952 25098 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-admin
2026-08-06 02:52:28,035 [I] DB FLAVOR :POSTGRES
2026-08-06 02:52:28,036 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:52:28,036 [I] Checking connection..
2026-08-06 02:52:28,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:52:28,290 [I] Checking connection passed.
2026-08-06 02:52:28,290 [I] --------- Verifying version history table ---------
2026-08-06 02:52:28,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:52:28,559 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:52:28,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:52:28,828 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:52:28,828 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:52:28,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:52:29.065 UTC [9716] WARNING: there is no transaction in progress
2026-08-06 02:52:29,074 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:52:29,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:52:29,347 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:52:29,347 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:52:29,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:52:29,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:52:29,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3kk6.c.gcp-upstream.internal','N') ;"
2026-08-06 02:52:30,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:52:30,452 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:52:30,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:52:30,722 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:52:30,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:52:30,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:52:31,257 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:52:31,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:52:32.309 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.327 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.345 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.361 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.379 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.402 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.473 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.481 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.497 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.509 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.520 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.536 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.547 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.554 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.562 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:32.570 UTC [9884] WARNING: there is no transaction in progress
2026-08-06 02:52:33,253 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:52:33,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:52:33,488 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:52:33,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:52:33,749 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:52:33,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:52:34,024 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:52:34,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:52:34,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:52:34,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3kk6.c.gcp-upstream.internal';"
2026-08-06 02:52:34,812 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:52:34,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:52:35,074 [I] DB_PATCHES have already been applied
I20260806 02:52:35.080942 25098 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGNnxjX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:52:35.081020 25098 mini_ranger.cc:193] Using host: 127.24.130.148
I20260806 02:52:35.085948 25098 mini_ranger.cc:227] Ranger admin URL: http://127.24.130.148:34965
Aug 06, 2026 2:52:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:52:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:52:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:52:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:52:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:52:36.402248 25098 mini_ranger.cc:160] Time spent starting Ranger: real 9.161s user 0.001s sys 0.012s
Aug 06, 2026 2:52:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:52:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 2:52:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:52:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:52:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:52:59.577116 25098 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.130.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.130.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/rangerkms_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.130.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/rangerkms_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.130.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.130.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.130.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/HTTP_127.24.130.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:52:59 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984779, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:52:59.683003 25098 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:52:59.695572 25098 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:53:00.656599 25098 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:53:00.656699 25098 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms
I20260806 02:53:00.656724 25098 mini_ranger_kms.cc:227] Using postgres at 127.24.130.148:35947
2026-08-06 02:53:00,855 [I] DB FLAVOR :POSTGRES
2026-08-06 02:53:00,855 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:53:00,855 [I] Checking connection
2026-08-06 02:53:00,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:53:01,113 [I] connection success
2026-08-06 02:53:01,113 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:53:01,114 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:53:01,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:53:01,368 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:53:01,368 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:53:01,368 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:53:01,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.130.148:35947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:53:01,650 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:53:02.224260 25098 mini_ranger_kms.cc:310] Created kms service
I20260806 02:53:02.458598 25098 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:53:02.510952 25098 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:53:02 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984779, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:02.955091 25098 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:53:02.955268 25098 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGNnxjX/build/debug/bin/postgresql.jar:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGNnxjX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:53:02.955289 25098 mini_ranger_kms.cc:240] Using host: 127.24.130.148
I20260806 02:53:02.960485 25098 mini_ranger_kms.cc:276] Ranger KMS PID: 10142
I20260806 02:53:02.960592 25098 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.130.148:35917
Aug 06, 2026 2:53:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:53:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:53:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:53:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:53:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:53:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:53:05 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.130.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:05.710449 25098 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.027s user 0.011s sys 0.172s
I20260806 02:53:05.710599 25098 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:53:05.710700 25098 mini_ranger_kms.cc:401] 127.24.130.148:35917/kms/v1/keys
Aug 06 02:53:05 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.130.148@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
Aug 06 02:53:06 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.130.148@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
Aug 06 02:53:08 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:53:08.822245 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38499
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:38499
--ranger_config_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:08.965920 10216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:08.966240 10216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:08.966282 10216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:08.970518 10216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:08.970602 10216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:08.970623 10216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:08.970640 10216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:08.970717 10216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:08.976652 10216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.130.148:35917/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:38499
--ranger_config_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:08.977887 10216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:08.979089 10216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:08.988811 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:08.989068 10216 server_base.cc:1034] running on GCE node
W20260806 02:53:08.988790 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:08.988819 10226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:08.990011 10216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:08.991271 10216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:08.992471 10216 hybrid_clock.cc:648] HybridClock initialized: now 1785984788992458 us; error 39 us; skew 500 ppm
Aug 06 02:53:08 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984788, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:08.998394 10216 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:08.999513 10216 webserver.cc:460] Webserver started at http://127.24.130.190:45977/ using document root <none> and password file <none>
I20260806 02:53:09.000110 10216 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:09.000172 10216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:09.000419 10216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:53:09 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984788, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:09.064407 10216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "adbfffc272114d6cb67ae86950dea64a"
format_stamp: "Formatted at 2026-08-06 02:53:09 on dist-test-slave-3kk6"
server_key: "3de7927eed1ae3cd9e94b26dc404d75a"
server_key_iv: "15423c11054d580956ded45f3d3b0cfa"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:09.065135 10216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "adbfffc272114d6cb67ae86950dea64a"
format_stamp: "Formatted at 2026-08-06 02:53:09 on dist-test-slave-3kk6"
server_key: "3de7927eed1ae3cd9e94b26dc404d75a"
server_key_iv: "15423c11054d580956ded45f3d3b0cfa"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:09.069453 10216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:53:09.084635 10236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:09.086120 10216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:09.086288 10216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "adbfffc272114d6cb67ae86950dea64a"
format_stamp: "Formatted at 2026-08-06 02:53:09 on dist-test-slave-3kk6"
server_key: "3de7927eed1ae3cd9e94b26dc404d75a"
server_key_iv: "15423c11054d580956ded45f3d3b0cfa"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:09.086447 10216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:09.109951 10216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:09.113620 10216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:09.113862 10216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:09.123415 10216 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:38499
I20260806 02:53:09.123445 10319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:38499 every 8 connection(s)
I20260806 02:53:09.124644 10216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:09.127911 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:09.133674 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 10216
I20260806 02:53:09.133841 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:53:09.134155 25098 external_mini_cluster.cc:1428] Setting key 17cdb854c730c9e7b4be9847ee2efd70
I20260806 02:53:09.134266 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Bootstrap starting.
I20260806 02:53:09.137466 10320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:09.138468 10320 log.cc:824] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:09.141134 10320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: No bootstrap required, opened a new log
Aug 06 02:53:09 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:09.146330 10320 raft_consensus.cc:348] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:09.146693 10320 raft_consensus.cc:374] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:09.146811 10320 raft_consensus.cc:729] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbfffc272114d6cb67ae86950dea64a, State: Initialized, Role: FOLLOWER
I20260806 02:53:09.147346 10320 consensus_queue.cc:260] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:09.147535 10320 raft_consensus.cc:388] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:09.147634 10320 raft_consensus.cc:482] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:09.147787 10320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:09.149014 10320 raft_consensus.cc:504] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:09.149495 10320 leader_election.cc:302] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adbfffc272114d6cb67ae86950dea64a; no voters:
I20260806 02:53:09.149909 10320 leader_election.cc:288] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:09.150025 10325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:09.150350 10325 raft_consensus.cc:686] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 LEADER]: Becoming Leader. State: Replica: adbfffc272114d6cb67ae86950dea64a, State: Running, Role: LEADER
I20260806 02:53:09.150759 10325 consensus_queue.cc:237] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:09.150908 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:09.135842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.24.130.190:38499)
0806 02:53:09.136391 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:09.136411 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:09.136457 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:53:09.137436 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:09.137462 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:09.137933 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:09.138344 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:09.139332 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.140454 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.141457 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.141985 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.142176 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:09.145032 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:09.145039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:09.145054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:09.145083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:09.147187 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:09.147773 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:09.147784 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:09.147791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:09.147957 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:09.148277 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:09.148282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:09.148283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:09.148732 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:09.148957 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:09.149283 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 02:53:09.149795 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":179,"threads_started":1}
I20260806 02:53:09.151357 10320 sys_catalog.cc:564] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:09.152170 10325 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adbfffc272114d6cb67ae86950dea64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } } }
I20260806 02:53:09.152382 10325 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:09.152658 10325 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader adbfffc272114d6cb67ae86950dea64a. Latest consensus state: current_term: 1 leader_uuid: "adbfffc272114d6cb67ae86950dea64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } } }
I20260806 02:53:09.152812 10325 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:09.153515 10334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:09.157001 10334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:09.157457 10320 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:53:09.168561 10334 catalog_manager.cc:1347] Generated new cluster ID: 28bdfe49bff94836b4791965e3f47297
I20260806 02:53:09.168738 10334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:09.182771 10334 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:09.184859 10334 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:09.198313 10334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Generated new TSK 0
I20260806 02:53:09.199465 10334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:53:09.985786 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38499
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:10.147764 10382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:10.148120 10382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:10.148211 10382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:10.152741 10382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:10.152845 10382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:10.152940 10382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:53:10.159880 10382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.130.148:35917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:10.161749 10382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:10.163291 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:10.177861 10394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:10.177861 10390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:10.178442 10382 server_base.cc:1034] running on GCE node
W20260806 02:53:10.178525 10391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:10.179088 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:10.180092 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:10.181316 10382 hybrid_clock.cc:648] HybridClock initialized: now 1785984790181242 us; error 92 us; skew 500 ppm
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:10.185434 10382 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:53:10.186492 10382 webserver.cc:460] Webserver started at http://127.24.130.129:34875/ using document root <none> and password file <none>
I20260806 02:53:10.187213 10382 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:10.187325 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:10.187589 10382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:10.207137 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "d624ce3313fe409bbaf0125ea71b62e1"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "70df742b67f7fc491f4f6b41cf074c4f"
server_key_iv: "d4d27ca1cfc742b70f1105b4b6110da0"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:10.208125 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "d624ce3313fe409bbaf0125ea71b62e1"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "70df742b67f7fc491f4f6b41cf074c4f"
server_key_iv: "d4d27ca1cfc742b70f1105b4b6110da0"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:10.214537 10382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 02:53:10.236233 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:10.238363 10382 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260806 02:53:10.238545 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "d624ce3313fe409bbaf0125ea71b62e1"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "70df742b67f7fc491f4f6b41cf074c4f"
server_key_iv: "d4d27ca1cfc742b70f1105b4b6110da0"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:10.238862 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:10.271323 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:10.275597 10382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:10.275921 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:10.276808 10382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:10.277994 10382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:10.278107 10382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:10.278234 10382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:10.278335 10382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:10.292946 10382 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:43933
I20260806 02:53:10.293004 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:43933 every 8 connection(s)
I20260806 02:53:10.294631 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:10.301580 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 10382
I20260806 02:53:10.301726 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:10.302032 25098 external_mini_cluster.cc:1428] Setting key 5af55e014dddd6633565416be52d6665
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:10.333511 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.298825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:48739 (local address 127.24.130.190:38499)
0806 02:53:10.311621 (+ 12796us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:10.311627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:10.311655 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:53:10.311744 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:10.311748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:10.311813 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:10.311983 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:10.318928 (+ 6945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.319929 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.321869 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.322232 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.322327 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.327987 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:10.327994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:10.328000 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:10.328044 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:10.330271 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:10.331249 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:10.331253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:10.331255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:10.331385 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:10.331952 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:10.331956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:10.331958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:10.332229 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:10.332324 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:10.333127 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 02:53:10.333289 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12670,"thread_start_us":98,"threads_started":1}
I20260806 02:53:10.335883 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.297583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38499 (local address 127.24.130.129:48739)
0806 02:53:10.303054 (+ 5471us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:10.303141 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:10.304273 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:10.317365 (+ 13092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:10.317379 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:10.317834 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:10.318612 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.318636 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.320169 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.320173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.321643 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.321658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.322609 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.322613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.322895 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.325352 (+ 2457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:10.325381 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:10.327660 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:10.330519 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:10.330527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:10.330552 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:10.331091 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:10.331530 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:10.331534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:10.331535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:10.331812 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:10.332418 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:10.332425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:10.332811 (+ 386us) client_negotiation.cc:769] Sending connection context
0806 02:53:10.334112 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0806 02:53:10.334560 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5171,"thread_start_us":176,"threads_started":1}
I20260806 02:53:10.337965 10571 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38499
I20260806 02:53:10.338243 10571 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:10.338986 10571 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:10.341288 10265 ts_manager.cc:194] Registered new tserver with Master: d624ce3313fe409bbaf0125ea71b62e1 (127.24.130.129:43933)
I20260806 02:53:10.343098 10265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:48739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:53:10.358139 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38499
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:53:10.532159 10578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:10.532609 10578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:10.532689 10578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:10.539392 10578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:10.539598 10578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:10.539710 10578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:53:10.548549 10578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.130.148:35917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:10.550340 10578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:10.552008 10578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:10.566144 10588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:10.566447 10586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:10.566236 10585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:10.567750 10578 server_base.cc:1034] running on GCE node
I20260806 02:53:10.568347 10578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:10.569345 10578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:10.570586 10578 hybrid_clock.cc:648] HybridClock initialized: now 1785984790570523 us; error 82 us; skew 500 ppm
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:10.574908 10578 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:53:10.576192 10578 webserver.cc:460] Webserver started at http://127.24.130.130:33839/ using document root <none> and password file <none>
I20260806 02:53:10.577003 10578 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:10.577104 10578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:10.577358 10578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:10.594383 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "b3611ddf60fe4675bd95b13c7bf44080"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "75ca536ad398483fba88ec53581097d7"
server_key_iv: "5aea55b0c54fc09458daf83ebe1834dd"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:10.595682 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "b3611ddf60fe4675bd95b13c7bf44080"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "75ca536ad398483fba88ec53581097d7"
server_key_iv: "5aea55b0c54fc09458daf83ebe1834dd"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:10.603119 10578 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 02:53:10.628028 10597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:10.630424 10578 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260806 02:53:10.630703 10578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "b3611ddf60fe4675bd95b13c7bf44080"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "75ca536ad398483fba88ec53581097d7"
server_key_iv: "5aea55b0c54fc09458daf83ebe1834dd"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:10.630966 10578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:10.647660 10578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:10.652070 10578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:10.652315 10578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:10.653095 10578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:10.654265 10578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:10.654335 10578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:10.654389 10578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:10.654462 10578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:10.666822 10578 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:40047
I20260806 02:53:10.668287 10578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:10.668247 10753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:40047 every 8 connection(s)
I20260806 02:53:10.671326 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 10578
I20260806 02:53:10.671485 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:10.671841 25098 external_mini_cluster.cc:1428] Setting key 5fe07940f9b2621590a2c679723abdfd
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:10.688211 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.670771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:44769 (local address 127.24.130.190:38499)
0806 02:53:10.670958 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:10.670963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:10.672860 (+ 1897us) server_negotiation.cc:407] Connection header received
0806 02:53:10.674242 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:10.674246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:10.674304 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:10.674362 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:10.676275 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.677115 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.679022 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.679404 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.679496 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.683128 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:10.683133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:10.683136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:10.683168 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:10.685175 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:10.685948 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:10.685952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:10.685954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:10.686054 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:10.686541 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:10.686546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:10.686548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:10.686830 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:10.686921 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:10.687862 (+ 941us) server_negotiation.cc:299] Negotiation successful
0806 02:53:10.688031 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:53:10.690536 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.671320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38499 (local address 127.24.130.130:44769)
0806 02:53:10.672674 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:10.672744 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:10.673950 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:10.674846 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:10.674855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:10.675295 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:10.676054 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.676077 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.677312 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.677316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.678860 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.678873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.679743 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.679747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.679974 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.680824 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:10.680857 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:10.682877 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:10.685372 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:10.685380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:10.685401 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:10.685808 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:10.686171 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:10.686174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:10.686176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:10.686405 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:10.687146 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:10.687154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:10.687541 (+ 387us) client_negotiation.cc:769] Sending connection context
0806 02:53:10.688881 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0806 02:53:10.689325 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":169,"threads_started":1}
I20260806 02:53:10.694725 10754 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38499
I20260806 02:53:10.695055 10754 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:10.695793 10754 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:10.697259 10265 ts_manager.cc:194] Registered new tserver with Master: b3611ddf60fe4675bd95b13c7bf44080 (127.24.130.130:40047)
I20260806 02:53:10.698040 10265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:44769
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:53:10.739886 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38499
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:10 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984790, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
W20260806 02:53:10.975922 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:10.977052 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:10.977209 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:10.983738 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:10.983934 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:10.984091 10762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:53:10.992787 10762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.130.148:35917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:10.994704 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:10.996296 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:11.012002 10775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:11.013800 10762 server_base.cc:1034] running on GCE node
W20260806 02:53:11.014796 10772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:11.015046 10770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:11.015672 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:11.016618 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:11.019896 10762 hybrid_clock.cc:648] HybridClock initialized: now 1785984791019755 us; error 154 us; skew 500 ppm
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:11.024289 10762 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:53:11.025596 10762 webserver.cc:460] Webserver started at http://127.24.130.131:33009/ using document root <none> and password file <none>
I20260806 02:53:11.026430 10762 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:11.026567 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:11.026895 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:11.044255 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "d373e3fee5c24cc79b433f11ada9b11c"
format_stamp: "Formatted at 2026-08-06 02:53:11 on dist-test-slave-3kk6"
server_key: "17c4dd1b19aae636a509617d6b0c6ba0"
server_key_iv: "2081321ce7ece386be5cd5a541cdeb62"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:11.050880 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "d373e3fee5c24cc79b433f11ada9b11c"
format_stamp: "Formatted at 2026-08-06 02:53:11 on dist-test-slave-3kk6"
server_key: "17c4dd1b19aae636a509617d6b0c6ba0"
server_key_iv: "2081321ce7ece386be5cd5a541cdeb62"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:11.072820 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.001s sys 0.004s
I20260806 02:53:11.099088 10783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.101065 10762 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260806 02:53:11.101351 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "d373e3fee5c24cc79b433f11ada9b11c"
format_stamp: "Formatted at 2026-08-06 02:53:11 on dist-test-slave-3kk6"
server_key: "17c4dd1b19aae636a509617d6b0c6ba0"
server_key_iv: "2081321ce7ece386be5cd5a541cdeb62"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:11.101614 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:11.160820 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:11.164996 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:11.165310 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:11.166184 10762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:11.167464 10762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:11.167574 10762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.167676 10762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:11.167757 10762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.180634 10762 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:44487
I20260806 02:53:11.180764 10939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:44487 every 8 connection(s)
I20260806 02:53:11.181999 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:11.190383 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 10762
I20260806 02:53:11.190565 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:11.190914 25098 external_mini_cluster.cc:1428] Setting key 3deef7313380cc1c8f234b574126418a
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:11.203274 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.184352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:37717 (local address 127.24.130.190:38499)
0806 02:53:11.184490 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:11.184495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:11.185673 (+ 1178us) server_negotiation.cc:407] Connection header received
0806 02:53:11.187077 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:11.187081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:11.187145 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:11.187203 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:11.189020 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.189907 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.193894 (+ 3987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.194260 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.194379 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.198011 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:11.198016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:11.198019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:11.198055 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:11.200214 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:11.201099 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:11.201104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:11.201106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:11.201213 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:11.201747 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:11.201750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:11.201752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:11.202020 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:11.202127 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:11.202954 (+ 827us) server_negotiation.cc:299] Negotiation successful
0806 02:53:11.203111 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:53:11.204828 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.184866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38499 (local address 127.24.130.131:37717)
0806 02:53:11.185478 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:11.185541 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:11.186773 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:11.187659 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:11.187669 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:11.188084 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:11.188805 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.188828 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.190935 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.190939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.193735 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.193750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.194469 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.194474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.194750 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.195601 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:11.195627 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:11.197730 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:11.200409 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:11.200418 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:11.200439 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:11.200924 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:11.201345 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:11.201349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:11.201351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:11.201600 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:11.202169 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:11.202179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:11.202587 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 02:53:11.202874 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 02:53:11.203294 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":158,"threads_started":1}
I20260806 02:53:11.206903 10940 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38499
I20260806 02:53:11.207207 10940 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:11.207984 10940 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:11.209218 10265 ts_manager.cc:194] Registered new tserver with Master: d373e3fee5c24cc79b433f11ada9b11c (127.24.130.131:44487)
I20260806 02:53:11.210001 10265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:37717
I20260806 02:53:11.217984 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:53:11.218816 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 10382
I20260806 02:53:11.235608 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 10578
I20260806 02:53:11.249295 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 10762
I20260806 02:53:11.263541 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 10216
I20260806 02:53:11.281011 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38499
--webserver_interface=127.24.130.190
--webserver_port=45977
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:38499
--ranger_config_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:11.416728 10943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:11.417129 10943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:11.417215 10943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:11.421566 10943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:11.421705 10943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:11.421751 10943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:11.421792 10943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:11.421823 10943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:11.427968 10943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:38499
--ranger_config_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:11.429215 10943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:11.430346 10943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:11.439431 10950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:11.439517 10953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:11.439713 10943 server_base.cc:1034] running on GCE node
W20260806 02:53:11.439435 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:11.440536 10943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:11.441825 10943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:11.443035 10943 hybrid_clock.cc:648] HybridClock initialized: now 1785984791443001 us; error 56 us; skew 500 ppm
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:11.446065 10943 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:11.447136 10943 webserver.cc:460] Webserver started at http://127.24.130.190:45977/ using document root <none> and password file <none>
I20260806 02:53:11.447717 10943 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:11.447840 10943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:11.529098 10943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:11.531834 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.533432 10943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:11.533646 10943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "adbfffc272114d6cb67ae86950dea64a"
format_stamp: "Formatted at 2026-08-06 02:53:09 on dist-test-slave-3kk6"
server_key: "3de7927eed1ae3cd9e94b26dc404d75a"
server_key_iv: "15423c11054d580956ded45f3d3b0cfa"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:11.534308 10943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:11.549817 10943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:11.553251 10943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:11.553519 10943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:11.561704 10943 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:38499
I20260806 02:53:11.561739 11041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:38499 every 8 connection(s)
I20260806 02:53:11.563043 10943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:11.567871 11042 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:53:11.569818 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Bootstrap starting.
I20260806 02:53:11.572494 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 10943
I20260806 02:53:11.573030 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:43933
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:38499
--webserver_port=34875
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:11.587383 11042 log.cc:824] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:11.594872 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:11.595352 11042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Bootstrap complete.
I20260806 02:53:11.599831 11042 raft_consensus.cc:348] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:11.600298 11042 raft_consensus.cc:729] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbfffc272114d6cb67ae86950dea64a, State: Initialized, Role: FOLLOWER
I20260806 02:53:11.600873 11042 consensus_queue.cc:260] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:11.601029 11042 raft_consensus.cc:388] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:11.601122 11042 raft_consensus.cc:482] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:11.601248 11042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:53:11.603437 11042 raft_consensus.cc:504] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:11.603801 11042 leader_election.cc:302] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adbfffc272114d6cb67ae86950dea64a; no voters:
I20260806 02:53:11.604223 11042 leader_election.cc:288] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:53:11.604315 11048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:53:11.604581 11048 raft_consensus.cc:686] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [term 2 LEADER]: Becoming Leader. State: Replica: adbfffc272114d6cb67ae86950dea64a, State: Running, Role: LEADER
I20260806 02:53:11.604971 11048 consensus_queue.cc:237] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } }
I20260806 02:53:11.605572 11042 sys_catalog.cc:564] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:11.606379 11049 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "adbfffc272114d6cb67ae86950dea64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } } }
I20260806 02:53:11.606523 11049 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:11.607275 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader adbfffc272114d6cb67ae86950dea64a. Latest consensus state: current_term: 2 leader_uuid: "adbfffc272114d6cb67ae86950dea64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfffc272114d6cb67ae86950dea64a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38499 } } }
I20260806 02:53:11.607398 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:11.607781 11057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:11.608270 11042 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:53:11.614687 11057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:11.615615 11057 catalog_manager.cc:1259] Loaded cluster ID: 28bdfe49bff94836b4791965e3f47297
I20260806 02:53:11.615684 11057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:11.618325 11057 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:11.619185 11057 catalog_manager.cc:5879] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Loaded TSK: 0
I20260806 02:53:11.620564 11057 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:53:11.706857 11046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:11.707389 11046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:11.707494 11046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:11.712365 11046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:11.712481 11046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:11.712596 11046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:53:11.718848 11046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:43933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=34875
--tserver_master_addrs=127.24.130.190:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:11.720556 11046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:11.722056 11046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:11.734009 11094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:11.734015 11096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:11.734478 11093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:11.735183 11046 server_base.cc:1034] running on GCE node
I20260806 02:53:11.735661 11046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:11.736403 11046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:11.737614 11046 hybrid_clock.cc:648] HybridClock initialized: now 1785984791737579 us; error 51 us; skew 500 ppm
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:11.741117 11046 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:53:11.742578 11046 webserver.cc:460] Webserver started at http://127.24.130.129:34875/ using document root <none> and password file <none>
I20260806 02:53:11.743359 11046 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:11.743489 11046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:11.770108 11046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:53:11.773216 11108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.775481 11046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:53:11.775756 11046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "d624ce3313fe409bbaf0125ea71b62e1"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "70df742b67f7fc491f4f6b41cf074c4f"
server_key_iv: "d4d27ca1cfc742b70f1105b4b6110da0"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:11.776594 11046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:11.800284 11046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:11.803762 11046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:11.804020 11046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:11.804845 11046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:11.806041 11046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:11.806156 11046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.806311 11046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:11.806416 11046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:11.818413 11046 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:43933
I20260806 02:53:11.818434 11272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:43933 every 8 connection(s)
I20260806 02:53:11.820541 11046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:11.823971 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 11046
I20260806 02:53:11.824542 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:40047
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:38499
--webserver_port=33839
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:11 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:11.851923 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.822978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:45657 (local address 127.24.130.190:38499)
0806 02:53:11.825963 (+ 2985us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:11.825986 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:11.826034 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:53:11.826352 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:11.826366 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:11.826813 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:11.827171 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:11.828892 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.829945 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.831453 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.832171 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.832349 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.835655 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:11.835663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:11.835684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:11.835724 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:11.838016 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:11.847783 (+ 9767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:11.847793 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:11.847801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:11.848025 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:11.848600 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:11.848604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:11.848607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:11.849189 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:11.849414 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:11.850269 (+ 855us) server_negotiation.cc:299] Negotiation successful
0806 02:53:11.850684 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2809,"thread_start_us":118,"threads_started":1}
I20260806 02:53:11.854075 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.823148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38499 (local address 127.24.130.129:45657)
0806 02:53:11.824329 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:11.824378 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:11.825431 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:11.827678 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:11.827687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:11.828067 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:11.828698 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.828716 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.830105 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.830108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.831296 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.831307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.832400 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.832405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.832615 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.833430 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:11.833485 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:11.835396 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:11.843734 (+ 8338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:11.843742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:11.843763 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:11.844238 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:11.848194 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:11.848199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:11.848205 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:11.848459 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:11.849498 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:11.849509 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:11.849857 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 02:53:11.852771 (+ 2914us) client_negotiation.cc:240] Negotiation successful
0806 02:53:11.853138 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":129,"threads_started":1}
I20260806 02:53:11.857510 11273 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38499
I20260806 02:53:11.857810 11273 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:11.858443 11273 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
W20260806 02:53:11.860872 11273 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:38499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:53:11.860999 11273 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:11.861186 11273 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:11.862083 11273 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:11.862257 11273 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
W20260806 02:53:11.863060 11273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:53:11.984660 11279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:11.985008 11279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:11.985118 11279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:11.989671 11279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:11.989785 11279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:11.989907 11279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:53:11.996001 11279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:40047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=33839
--tserver_master_addrs=127.24.130.190:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:11.997285 11279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:11.998409 11279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:12.010175 11288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:12.010175 11285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:12.010588 11279 server_base.cc:1034] running on GCE node
W20260806 02:53:12.010175 11290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:12.011189 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:12.012151 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:12.013371 11279 hybrid_clock.cc:648] HybridClock initialized: now 1785984792013338 us; error 55 us; skew 500 ppm
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:12.017325 11279 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:53:12.018728 11279 webserver.cc:460] Webserver started at http://127.24.130.130:33839/ using document root <none> and password file <none>
I20260806 02:53:12.019533 11279 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:12.019654 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:12.045579 11279 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.000s
I20260806 02:53:12.049326 11303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:12.051203 11279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:53:12.051379 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "b3611ddf60fe4675bd95b13c7bf44080"
format_stamp: "Formatted at 2026-08-06 02:53:10 on dist-test-slave-3kk6"
server_key: "75ca536ad398483fba88ec53581097d7"
server_key_iv: "5aea55b0c54fc09458daf83ebe1834dd"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:12.052263 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:12.082926 11279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:12.087450 11279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:12.087782 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:12.088670 11279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:12.090101 11279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:12.090221 11279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:12.090327 11279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:12.090407 11279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:12.106159 11279 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:40047
I20260806 02:53:12.108258 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:12.109797 11470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:40047 every 8 connection(s)
I20260806 02:53:12.117142 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 11279
I20260806 02:53:12.117687 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.130.148:35917/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:44487
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:38499
--webserver_port=33009
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:32889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:12.128877 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.113287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:35849 (local address 127.24.130.190:38499)
0806 02:53:12.113451 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:12.113457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:12.114331 (+ 874us) server_negotiation.cc:407] Connection header received
0806 02:53:12.115592 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:12.115596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:12.115655 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:12.115708 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:12.117551 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.118406 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.120162 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.120579 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.120665 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.124063 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:12.124067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:12.124070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:12.124103 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:12.126067 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:12.126809 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:12.126815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:12.126820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:12.126922 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:12.127408 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:12.127412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:12.127414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:12.127676 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:12.127775 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:12.128535 (+ 760us) server_negotiation.cc:299] Negotiation successful
0806 02:53:12.128693 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:53:12.130975 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.112046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38499 (local address 127.24.130.130:35849)
0806 02:53:12.114137 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:12.114184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:12.115283 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:12.116141 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:12.116150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:12.116544 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:12.117341 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.117372 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.118579 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.118582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.120009 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.120021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.120908 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.120912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.121119 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.121905 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:12.121927 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:12.123822 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:12.126217 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:12.126224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:12.126243 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:12.126629 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:12.127044 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:12.127047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:12.127049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:12.127270 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:12.127850 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:12.127857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:12.128219 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 02:53:12.129421 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0806 02:53:12.129808 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"thread_start_us":196,"threads_started":1}
I20260806 02:53:12.132786 11472 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38499
I20260806 02:53:12.133064 11472 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.133698 11472 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
W20260806 02:53:12.135684 11472 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:38499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:53:12.135810 11472 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.135988 11472 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:12.137024 11472 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.137209 11472 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
W20260806 02:53:12.138203 11472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:53:12.278756 11478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:12.279304 11478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:12.279405 11478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:12.285642 11478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:12.285800 11478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:12.285938 11478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:53:12.294525 11478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:32889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.130.148:35917/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:44487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=33009
--tserver_master_addrs=127.24.130.190:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:12.296108 11478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:12.297572 11478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:12.310786 11486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:12.310787 11488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:12.311309 11490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:12.311664 11478 server_base.cc:1034] running on GCE node
I20260806 02:53:12.312188 11478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:12.313066 11478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:12.314335 11478 hybrid_clock.cc:648] HybridClock initialized: now 1785984792314294 us; error 54 us; skew 500 ppm
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:12.318601 11478 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:53:12.319972 11478 webserver.cc:460] Webserver started at http://127.24.130.131:33009/ using document root <none> and password file <none>
I20260806 02:53:12.320861 11478 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:12.321038 11478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:12.343972 11478 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:53:12.347429 11502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:12.349228 11478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:53:12.349404 11478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "d373e3fee5c24cc79b433f11ada9b11c"
format_stamp: "Formatted at 2026-08-06 02:53:11 on dist-test-slave-3kk6"
server_key: "17c4dd1b19aae636a509617d6b0c6ba0"
server_key_iv: "2081321ce7ece386be5cd5a541cdeb62"
server_key_version: "kuduclusterkey@0"
I20260806 02:53:12.350226 11478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:12.380203 11478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:12.384527 11478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:12.384903 11478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:12.385793 11478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:12.387033 11478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:12.387140 11478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:12.387241 11478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:12.387303 11478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:12.402984 11478 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:44487
I20260806 02:53:12.403049 11659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:44487 every 8 connection(s)
I20260806 02:53:12.404520 11478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:12.409579 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 11478
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:12.424940 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.407064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:53287 (local address 127.24.130.190:38499)
0806 02:53:12.407228 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:12.407233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:12.408344 (+ 1111us) server_negotiation.cc:407] Connection header received
0806 02:53:12.410432 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:12.410437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:12.410508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:12.410572 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:12.412288 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.413130 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.414815 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.415047 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.415174 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.419660 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:12.419664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:12.419668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:12.419703 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:12.421565 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:12.422777 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:12.422781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:12.422783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:12.422889 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:12.423424 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:12.423429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:12.423432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:12.423692 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:12.423793 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:12.424669 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 02:53:12.424807 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:53:12.426543 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.407557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38499 (local address 127.24.130.131:53287)
0806 02:53:12.408125 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:12.408196 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:12.410116 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:12.411097 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:12.411106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:12.411475 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:12.412049 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.412066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.413339 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.413343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.414590 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.414603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.415251 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.415256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.415480 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.416323 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:12.416349 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:12.419419 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:12.422084 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:12.422092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:12.422114 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:12.422622 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:12.423018 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:12.423021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:12.423023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:12.423281 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:12.423876 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:12.423886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:12.424325 (+ 439us) client_negotiation.cc:769] Sending connection context
0806 02:53:12.425259 (+ 934us) client_negotiation.cc:240] Negotiation successful
0806 02:53:12.425603 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":137,"threads_started":1}
I20260806 02:53:12.428270 11660 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38499
I20260806 02:53:12.428550 11660 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.429216 11660 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
W20260806 02:53:12.431236 11660 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:38499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:53:12.431402 11660 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.431617 11660 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:12.432302 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.410465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.24.130.190:38499)
0806 02:53:12.410778 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:12.410783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:12.415620 (+ 4837us) server_negotiation.cc:407] Connection header received
0806 02:53:12.415798 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:12.415801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:12.415864 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:12.415974 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:12.417964 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.418833 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.420762 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.421992 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.422105 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.426941 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:12.426946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:12.426949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:12.426981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:12.428854 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:12.429433 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:12.429437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:12.429439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:12.429550 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:12.430185 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:12.430189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:12.430190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:12.430465 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:12.430737 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:12.431938 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0806 02:53:12.432086 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
I20260806 02:53:12.432618 11660 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.432894 11660 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
W20260806 02:53:12.434167 11660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 02:53:12 dist-test-slave-3kk6 krb5kdc[9565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:12.863386 11273 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:12.863795 11273 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:12.865330 10989 ts_manager.cc:194] Registered new tserver with Master: d624ce3313fe409bbaf0125ea71b62e1 (127.24.130.129:43933)
I20260806 02:53:12.867514 10989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:45657
I20260806 02:53:13.138440 11472 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:13.138860 11472 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:13.139858 10989 ts_manager.cc:194] Registered new tserver with Master: b3611ddf60fe4675bd95b13c7bf44080 (127.24.130.130:40047)
I20260806 02:53:13.140694 10989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:35849
Aug 06 02:53:13 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
I20260806 02:53:13.434479 11660 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:13.434859 11660 heartbeater.cc:511] Master 127.24.130.190:38499 requested a full tablet report, sending...
I20260806 02:53:13.435837 10989 ts_manager.cc:194] Registered new tserver with Master: d373e3fee5c24cc79b433f11ada9b11c (127.24.130.131:44487)
I20260806 02:53:13.436678 10989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:53287
I20260806 02:53:13.443094 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:53:13.455837 11694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.445027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.24.130.190:38499)
0806 02:53:13.445356 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.445361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.445777 (+ 416us) server_negotiation.cc:407] Connection header received
0806 02:53:13.445924 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.445928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.445986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.446097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:13.447011 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.447874 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.449091 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.449486 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.449601 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.451584 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:13.451589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:13.451592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:13.451625 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:13.453728 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:13.454283 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:13.454286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:13.454288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:13.454389 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:13.454739 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:13.454743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:13.454745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:13.454987 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:13.455125 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.455425 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.455644 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":123,"threads_started":1}
I20260806 02:53:13.460956 10989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:53:13.465714 10989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:53:13.487915 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.480080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43933 (local address 127.0.0.1:57586)
0806 02:53:13.480583 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.480642 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.480777 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.482678 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.482685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.482712 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:13.482877 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.482886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.484679 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.484683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.486082 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.486094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.487335 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.487339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.487463 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.487515 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 02:53:13.487613 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:53:13.487749 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":165,"threads_started":1}
I20260806 02:53:13.488778 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.481209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40047 (local address 127.0.0.1:53652)
0806 02:53:13.481485 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.481501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.481607 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.482875 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.482879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.482897 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:13.482994 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.482999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.485347 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.485351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.486466 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.486478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.488416 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.488421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.488511 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.488537 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:53:13.488582 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 02:53:13.488649 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
I20260806 02:53:13.489435 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.481002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.24.130.129:43933)
0806 02:53:13.482105 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.482113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.482142 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:53:13.482237 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.482242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.482468 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.483061 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:13.483286 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.484530 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.486260 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.487171 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.488979 (+ 1808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.489129 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.489183 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.489291 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":117,"threads_started":1}
I20260806 02:53:13.489530 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.481961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.24.130.130:40047)
0806 02:53:13.482404 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.482411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.482433 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:53:13.482505 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.482510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.482725 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.483138 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:13.484125 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.485214 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.487430 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.488275 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.489062 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.489256 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.489308 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.489397 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":93,"threads_started":1}
I20260806 02:53:13.492760 11348 tablet_service.cc:1467] Processing CreateTablet for tablet 8e30a68349014e0c9b0faff09a7d3c86 (DEFAULT_TABLE table=test-table [id=45746164c8f2417ab16106a57c105bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:13.492761 11186 tablet_service.cc:1467] Processing CreateTablet for tablet 8e30a68349014e0c9b0faff09a7d3c86 (DEFAULT_TABLE table=test-table [id=45746164c8f2417ab16106a57c105bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:13.493942 11348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e30a68349014e0c9b0faff09a7d3c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:13.493943 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e30a68349014e0c9b0faff09a7d3c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:13.495814 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.481546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:44487 (local address 127.0.0.1:51646)
0806 02:53:13.488108 (+ 6562us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.488124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.488219 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.491387 (+ 3168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.491391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.491411 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:13.491497 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.491503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.493184 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.493189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.494271 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.494283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.495382 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.495386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.495485 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.495531 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:53:13.495590 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 02:53:13.495664 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6497,"thread_start_us":83,"threads_started":1}
I20260806 02:53:13.496562 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.481739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.24.130.131:44487)
0806 02:53:13.490888 (+ 9149us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.490895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.490922 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:53:13.491001 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.491006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.491214 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.491654 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:13.491883 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.492942 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.494439 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.495232 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.496139 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.496272 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.496326 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.496419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9025,"thread_start_us":85,"threads_started":1}
I20260806 02:53:13.499538 11572 tablet_service.cc:1467] Processing CreateTablet for tablet 8e30a68349014e0c9b0faff09a7d3c86 (DEFAULT_TABLE table=test-table [id=45746164c8f2417ab16106a57c105bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:13.500672 11572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e30a68349014e0c9b0faff09a7d3c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:13.505133 11710 tablet_bootstrap.cc:492] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: Bootstrap starting.
I20260806 02:53:13.508260 11710 tablet_bootstrap.cc:654] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:13.509339 11710 log.cc:824] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:13.512518 11710 tablet_bootstrap.cc:492] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: No bootstrap required, opened a new log
I20260806 02:53:13.512754 11710 ts_tablet_manager.cc:1397] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 02:53:13.517230 11710 raft_consensus.cc:348] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.517735 11710 raft_consensus.cc:374] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:13.517864 11710 raft_consensus.cc:729] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d624ce3313fe409bbaf0125ea71b62e1, State: Initialized, Role: FOLLOWER
I20260806 02:53:13.518564 11710 consensus_queue.cc:260] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.519951 11710 ts_tablet_manager.cc:1428] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:53:13.520440 11273 heartbeater.cc:503] Master 127.24.130.190:38499 was elected leader, sending a full tablet report...
I20260806 02:53:13.525053 11712 tablet_bootstrap.cc:492] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Bootstrap starting.
I20260806 02:53:13.527976 11712 tablet_bootstrap.cc:654] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:13.529011 11712 log.cc:824] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:13.529594 11713 tablet_bootstrap.cc:492] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Bootstrap starting.
I20260806 02:53:13.532290 11713 tablet_bootstrap.cc:654] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:13.533843 11713 log.cc:824] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:13.537000 11712 tablet_bootstrap.cc:492] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: No bootstrap required, opened a new log
I20260806 02:53:13.539074 11712 ts_tablet_manager.cc:1397] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260806 02:53:13.540406 11713 tablet_bootstrap.cc:492] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: No bootstrap required, opened a new log
I20260806 02:53:13.541421 11713 ts_tablet_manager.cc:1397] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260806 02:53:13.546077 11713 raft_consensus.cc:348] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.546482 11713 raft_consensus.cc:374] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:13.550074 11712 raft_consensus.cc:348] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.550623 11712 raft_consensus.cc:374] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:13.550792 11712 raft_consensus.cc:729] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3611ddf60fe4675bd95b13c7bf44080, State: Initialized, Role: FOLLOWER
I20260806 02:53:13.551532 11712 consensus_queue.cc:260] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.552982 11712 ts_tablet_manager.cc:1428] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Time spent starting tablet: real 0.014s user 0.000s sys 0.006s
I20260806 02:53:13.553129 11713 raft_consensus.cc:729] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d373e3fee5c24cc79b433f11ada9b11c, State: Initialized, Role: FOLLOWER
I20260806 02:53:13.553822 11713 consensus_queue.cc:260] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.555135 11713 ts_tablet_manager.cc:1428] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260806 02:53:13.555696 11472 heartbeater.cc:503] Master 127.24.130.190:38499 was elected leader, sending a full tablet report...
I20260806 02:53:13.555996 11660 heartbeater.cc:503] Master 127.24.130.190:38499 was elected leader, sending a full tablet report...
W20260806 02:53:13.573172 11274 tablet.cc:2369] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:53:13 dist-test-slave-3kk6 krb5kdc[9565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for HTTP/127.24.130.148@KRBTEST.COM
W20260806 02:53:13.617908 11473 tablet.cc:2369] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:13.648491 11723 raft_consensus.cc:482] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:13.648785 11723 raft_consensus.cc:504] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.650051 11723 leader_election.cc:288] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d624ce3313fe409bbaf0125ea71b62e1 (127.24.130.129:43933), b3611ddf60fe4675bd95b13c7bf44080 (127.24.130.130:40047)
W20260806 02:53:13.656436 11664 tablet.cc:2369] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:13.663830 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.650855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:56849 (local address 127.24.130.130:40047)
0806 02:53:13.650981 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.650985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.658686 (+ 7701us) server_negotiation.cc:407] Connection header received
0806 02:53:13.659044 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.659049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.659109 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.659163 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:13.659669 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.660472 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.662245 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.662941 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.663059 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.663119 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.663556 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.663656 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:53:13.664541 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.650343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40047 (local address 127.24.130.131:56849)
0806 02:53:13.658546 (+ 8203us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.658563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.658769 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.659343 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.659346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.659367 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:13.659506 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.659512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.660643 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.660647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.662067 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.662079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.663254 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.663259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.663376 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.663406 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:53:13.664257 (+ 851us) client_negotiation.cc:240] Negotiation successful
0806 02:53:13.664354 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8106,"thread_start_us":124,"threads_started":1}
I20260806 02:53:13.665227 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.652369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:59765 (local address 127.24.130.129:43933)
0806 02:53:13.652530 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.652534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.655092 (+ 2558us) server_negotiation.cc:407] Connection header received
0806 02:53:13.655992 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.655996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.656049 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.656098 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:13.656919 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.657730 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.663635 (+ 5905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.664268 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.664377 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.664427 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.665007 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.665106 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:53:13.665737 11406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e30a68349014e0c9b0faff09a7d3c86" candidate_uuid: "d373e3fee5c24cc79b433f11ada9b11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3611ddf60fe4675bd95b13c7bf44080" is_pre_election: true
I20260806 02:53:13.666115 11406 raft_consensus.cc:2454] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d373e3fee5c24cc79b433f11ada9b11c in term 0.
I20260806 02:53:13.666739 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.650271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43933 (local address 127.24.130.131:59765)
0806 02:53:13.654975 (+ 4704us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.654994 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.655809 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.656250 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.656253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.656269 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:13.656429 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.656435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.657903 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.657988 (+ 85us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.663452 (+ 5464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.663466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.664614 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.664617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.664719 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.664743 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:53:13.666478 (+ 1735us) client_negotiation.cc:240] Negotiation successful
0806 02:53:13.666561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4614,"thread_start_us":158,"threads_started":1}
I20260806 02:53:13.666792 11503 leader_election.cc:302] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3611ddf60fe4675bd95b13c7bf44080, d373e3fee5c24cc79b433f11ada9b11c; no voters:
I20260806 02:53:13.667388 11723 raft_consensus.cc:2781] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:53:13.667485 11723 raft_consensus.cc:482] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:13.667537 11723 raft_consensus.cc:3037] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:13.668972 11210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e30a68349014e0c9b0faff09a7d3c86" candidate_uuid: "d373e3fee5c24cc79b433f11ada9b11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d624ce3313fe409bbaf0125ea71b62e1" is_pre_election: true
I20260806 02:53:13.669253 11723 raft_consensus.cc:504] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.669325 11210 raft_consensus.cc:2454] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d373e3fee5c24cc79b433f11ada9b11c in term 0.
I20260806 02:53:13.669705 11723 leader_election.cc:288] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [CANDIDATE]: Term 1 election: Requested vote from peers d624ce3313fe409bbaf0125ea71b62e1 (127.24.130.129:43933), b3611ddf60fe4675bd95b13c7bf44080 (127.24.130.130:40047)
I20260806 02:53:13.670326 11210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e30a68349014e0c9b0faff09a7d3c86" candidate_uuid: "d373e3fee5c24cc79b433f11ada9b11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d624ce3313fe409bbaf0125ea71b62e1"
I20260806 02:53:13.670367 11406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e30a68349014e0c9b0faff09a7d3c86" candidate_uuid: "d373e3fee5c24cc79b433f11ada9b11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3611ddf60fe4675bd95b13c7bf44080"
I20260806 02:53:13.670498 11210 raft_consensus.cc:3037] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:13.670504 11406 raft_consensus.cc:3037] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:13.671733 11210 raft_consensus.cc:2454] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d373e3fee5c24cc79b433f11ada9b11c in term 1.
I20260806 02:53:13.671921 11406 raft_consensus.cc:2454] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d373e3fee5c24cc79b433f11ada9b11c in term 1.
I20260806 02:53:13.672147 11504 leader_election.cc:302] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d373e3fee5c24cc79b433f11ada9b11c, d624ce3313fe409bbaf0125ea71b62e1; no voters:
I20260806 02:53:13.672525 11723 raft_consensus.cc:2781] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:13.672789 11723 raft_consensus.cc:686] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 1 LEADER]: Becoming Leader. State: Replica: d373e3fee5c24cc79b433f11ada9b11c, State: Running, Role: LEADER
I20260806 02:53:13.673202 11723 consensus_queue.cc:237] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } }
I20260806 02:53:13.677312 10991 catalog_manager.cc:5495] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c reported cstate change: term changed from 0 to 1, leader changed from <none> to d373e3fee5c24cc79b433f11ada9b11c (127.24.130.131). New cstate: current_term: 1 leader_uuid: "d373e3fee5c24cc79b433f11ada9b11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d373e3fee5c24cc79b433f11ada9b11c" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44487 } health_report { overall_health: HEALTHY } } }
I20260806 02:53:13.722980 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.717334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.24.130.131:44487)
0806 02:53:13.717488 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.717493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.717509 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:53:13.717613 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.717615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.717684 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.717805 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:53:13.718233 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.719065 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.720414 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.720759 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.720866 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.721341 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:53:13.721963 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:53:13.722092 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:13.722616 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 02:53:13.722728 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:53:13.736723 11406 raft_consensus.cc:1261] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 1 FOLLOWER]: Refusing update from remote peer d373e3fee5c24cc79b433f11ada9b11c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:13.737720 11729 consensus_queue.cc:1035] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3611ddf60fe4675bd95b13c7bf44080" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:13.745313 11210 raft_consensus.cc:1261] T 8e30a68349014e0c9b0faff09a7d3c86 P d624ce3313fe409bbaf0125ea71b62e1 [term 1 FOLLOWER]: Refusing update from remote peer d373e3fee5c24cc79b433f11ada9b11c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:13.746258 11729 consensus_queue.cc:1035] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d624ce3313fe409bbaf0125ea71b62e1" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:13.752321 11732 mvcc.cc:205] Tried to move back new op lower bound from 7315393715132989440 to 7315393714889814016. Current Snapshot: MvccSnapshot[applied={T|T < 7315393715132989440}]
I20260806 02:53:13.752744 11733 mvcc.cc:205] Tried to move back new op lower bound from 7315393715132989440 to 7315393714889814016. Current Snapshot: MvccSnapshot[applied={T|T < 7315393715132989440}]
I20260806 02:53:13.784811 10989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:13.785036 10989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:13.788229 10989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adbfffc272114d6cb67ae86950dea64a: Sending DeleteTablet for 3 replicas of tablet 8e30a68349014e0c9b0faff09a7d3c86
I20260806 02:53:13.789175 11186 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e30a68349014e0c9b0faff09a7d3c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:13 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:57586
I20260806 02:53:13.790428 11348 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e30a68349014e0c9b0faff09a7d3c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:13 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:53652
I20260806 02:53:13.791546 11572 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e30a68349014e0c9b0faff09a7d3c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:13 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:51646
I20260806 02:53:13.795056 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 11046
I20260806 02:53:13.798904 11752 tablet_replica.cc:330] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: stopping tablet replica
I20260806 02:53:13.799314 11752 raft_consensus.cc:2229] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:13.799701 11752 raft_consensus.cc:2258] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:13.800134 11753 tablet_replica.cc:330] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: stopping tablet replica
I20260806 02:53:13.800441 11753 raft_consensus.cc:2229] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:53:13.800822 11753 raft_consensus.cc:2258] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:13.808367 11753 ts_tablet_manager.cc:1905] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:13.812000 11753 ts_tablet_manager.cc:1918] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:13.812229 11753 log.cc:1196] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/8e30a68349014e0c9b0faff09a7d3c86
I20260806 02:53:13.812384 11752 ts_tablet_manager.cc:1905] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:13.812709 11753 ts_tablet_manager.cc:1939] T 8e30a68349014e0c9b0faff09a7d3c86 P d373e3fee5c24cc79b433f11ada9b11c: Deleting consensus metadata
I20260806 02:53:13.814239 10967 catalog_manager.cc:4841] TS d373e3fee5c24cc79b433f11ada9b11c (127.24.130.131:44487): tablet 8e30a68349014e0c9b0faff09a7d3c86 (table test-table [id=45746164c8f2417ab16106a57c105bed]) successfully deleted
I20260806 02:53:13.820109 11752 ts_tablet_manager.cc:1918] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:13.820276 11752 log.cc:1196] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/8e30a68349014e0c9b0faff09a7d3c86
I20260806 02:53:13.820631 11752 ts_tablet_manager.cc:1939] T 8e30a68349014e0c9b0faff09a7d3c86 P b3611ddf60fe4675bd95b13c7bf44080: Deleting consensus metadata
I20260806 02:53:13.822088 10967 catalog_manager.cc:4841] TS b3611ddf60fe4675bd95b13c7bf44080 (127.24.130.130:40047): tablet 8e30a68349014e0c9b0faff09a7d3c86 (table test-table [id=45746164c8f2417ab16106a57c105bed]) successfully deleted
W20260806 02:53:13.822301 10968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:43933 (error 108)
I20260806 02:53:13.823503 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.823153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43933 (local address 127.0.0.1:57592)
0806 02:53:13.823302 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.823394 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:53:13.823678 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 11279
W20260806 02:53:13.848140 10968 catalog_manager.cc:4584] TS d624ce3313fe409bbaf0125ea71b62e1 (127.24.130.129:43933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e30a68349014e0c9b0faff09a7d3c86: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43933: connect: Connection refused (error 111)
I20260806 02:53:13.859103 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 11478
I20260806 02:53:13.878584 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 10943
I20260806 02:53:13.896597 25098 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:53:14.243167 25098 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:53:14.243304 25098 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:53:14.589543 25098 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:53:14.589 UTC [9614] LOG: received smart shutdown request
2026-08-06 02:53:14.594 UTC [9614] LOG: background worker "logical replication launcher" (PID 9625) exited with exit code 1
2026-08-06 02:53:14.595 UTC [9619] LOG: shutting down
2026-08-06 02:53:14.773 UTC [9614] LOG: database system is shut down
2026-08-06T02:53:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52839 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:14 dist-test-slave-3kk6 krb5kdc[11768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:14 dist-test-slave-3kk6 krb5kdc[11768](info): set up 2 sockets
Aug 06 02:53:14 dist-test-slave-3kk6 krb5kdc[11768](info): commencing operation
krb5kdc: starting...
W20260806 02:53:16.132387 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:53:16.266444 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39297
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39297,127.24.130.189:44399,127.24.130.188:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:16.385226 11787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:16.385566 11787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:16.385659 11787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:16.389816 11787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:16.389947 11787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:16.389986 11787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:16.390045 11787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:16.390084 11787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:16.395622 11787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39297,127.24.130.189:44399,127.24.130.188:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:16.396827 11787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:16.397907 11787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:16.406963 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:16.406958 11795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:16.407049 11787 server_base.cc:1034] running on GCE node
W20260806 02:53:16.406958 11794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:16.407887 11787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:16.409034 11787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:16.410238 11787 hybrid_clock.cc:648] HybridClock initialized: now 1785984796410206 us; error 47 us; skew 500 ppm
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:16.413434 11787 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:16.414431 11787 webserver.cc:460] Webserver started at http://127.24.130.190:40389/ using document root <none> and password file <none>
I20260806 02:53:16.415083 11787 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:16.415179 11787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:16.415441 11787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:16.417625 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "bc6d7b4f0c854fbd80b75aa116626a20"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "9e8f0523c1ba0e06c767a65516b7d2bc"
server_key_iv: "4ead78265ccb5af24e67ecc39422906c"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.418236 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "bc6d7b4f0c854fbd80b75aa116626a20"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "9e8f0523c1ba0e06c767a65516b7d2bc"
server_key_iv: "4ead78265ccb5af24e67ecc39422906c"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.422047 11787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:53:16.424551 11805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:16.425721 11787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:53:16.425881 11787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "bc6d7b4f0c854fbd80b75aa116626a20"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "9e8f0523c1ba0e06c767a65516b7d2bc"
server_key_iv: "4ead78265ccb5af24e67ecc39422906c"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.426000 11787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:16.443835 11787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:16.447471 11787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:16.447711 11787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:16.455502 11787 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:39297
I20260806 02:53:16.455830 11883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:39297 every 8 connection(s)
I20260806 02:53:16.456593 11787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:16.459565 11884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:16.463620 11884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.464700 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 11787
I20260806 02:53:16.464901 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:53:16.465189 25098 external_mini_cluster.cc:1428] Setting key b4a52f09eb90242ced4d8c7f3c9df896
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.189@KRBTEST.COM" created.
I20260806 02:53:16.480093 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.466221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.24.130.190:39297)
0806 02:53:16.466856 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.466871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.466981 (+ 110us) server_negotiation.cc:407] Connection header received
0806 02:53:16.468081 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.468086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.468271 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.468469 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.469395 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.470387 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.471480 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.471880 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.472063 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.474593 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.474600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.474614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.474643 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.476600 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.477161 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.477167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.477169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.477311 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.477695 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.477701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.477705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.478289 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.478445 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.478882 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.479218 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":494,"thread_start_us":172,"threads_started":1}
I20260806 02:53:16.480094 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.466219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39297 (local address 127.0.0.1:49264)
0806 02:53:16.466775 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.466835 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.467747 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.468468 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.468473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.468949 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.469215 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.469230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.470533 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.470536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.471345 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.471356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.472051 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.472055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.472173 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.472836 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.472854 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.474436 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.476750 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.476774 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.476785 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.477058 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.477438 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.477441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.477442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.477577 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.478416 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.478422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.478555 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.478779 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.479108 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":3,"thread_start_us":152,"threads_started":1}
I20260806 02:53:16.482824 11884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.483654 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.483383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59284)
0806 02:53:16.483511 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.483586 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:53:16.483911 11810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
I20260806 02:53:16.484831 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.484617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59286)
0806 02:53:16.484727 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.484778 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:53:16.485018 11884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:44399: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab.
Entry for principal kudu/127.24.130.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab.
I20260806 02:53:16.497973 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:44399
--webserver_interface=127.24.130.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab
--principal=kudu/127.24.130.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39297,127.24.130.189:44399,127.24.130.188:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:16.534315 11884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } attempt: 1
I20260806 02:53:16.535490 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.535114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59294)
0806 02:53:16.535316 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.535402 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:53:16.536422 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.536162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59304)
0806 02:53:16.536293 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.536355 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:53:16.536583 11884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:44399: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
I20260806 02:53:16.604761 11884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } attempt: 2
I20260806 02:53:16.605733 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.605415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59312)
0806 02:53:16.605582 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.605652 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:53:16.606531 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.606311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59324)
0806 02:53:16.606422 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.606479 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:53:16.606689 11884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:44399: Network error: Client connection negotiation failed: client connection to 127.24.130.189:44399: connect: Connection refused (error 111)
W20260806 02:53:16.619072 11897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:16.619400 11897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:16.619508 11897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:16.623637 11897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:16.623731 11897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:16.623770 11897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:16.623826 11897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:16.623865 11897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:16.629551 11897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39297,127.24.130.189:44399,127.24.130.188:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab
--principal=kudu/127.24.130.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.130.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:16.630820 11897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:16.631915 11897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:16.641116 11904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:16.641108 11905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:16.641242 11897 server_base.cc:1034] running on GCE node
W20260806 02:53:16.641108 11908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:16.642216 11897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:16.643637 11897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:16.644872 11897 hybrid_clock.cc:648] HybridClock initialized: now 1785984796644857 us; error 36 us; skew 500 ppm
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:16.648488 11897 init.cc:377] Logged in from keytab as kudu/127.24.130.189@KRBTEST.COM (short username kudu)
I20260806 02:53:16.649665 11897 webserver.cc:460] Webserver started at http://127.24.130.189:40327/ using document root <none> and password file <none>
I20260806 02:53:16.650336 11897 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:16.650447 11897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:16.650813 11897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:16.653234 11897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data/instance:
uuid: "42d00cc7e07c4285853e1f103c1bcb71"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "a6fb26bac893f301437bc37e0cef9555"
server_key_iv: "5378cbd9e0ec0b8d775726f0c281fc92"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.653914 11897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal/instance:
uuid: "42d00cc7e07c4285853e1f103c1bcb71"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "a6fb26bac893f301437bc37e0cef9555"
server_key_iv: "5378cbd9e0ec0b8d775726f0c281fc92"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.658433 11897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:53:16.661351 11919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:16.662789 11897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:16.662958 11897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal
uuid: "42d00cc7e07c4285853e1f103c1bcb71"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "a6fb26bac893f301437bc37e0cef9555"
server_key_iv: "5378cbd9e0ec0b8d775726f0c281fc92"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.663105 11897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:16.677786 11897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:16.681383 11897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:16.681636 11897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:16.689857 11897 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.189:44399
I20260806 02:53:16.690150 11992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.189:44399 every 8 connection(s)
I20260806 02:53:16.691049 11897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
I20260806 02:53:16.693948 11993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:16.696364 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 11897
I20260806 02:53:16.696537 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-1/wal/instance
I20260806 02:53:16.696857 25098 external_mini_cluster.cc:1428] Setting key 8cd10c90e2b9d92b6951e95426c5bf7f
I20260806 02:53:16.697882 11884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } attempt: 3
I20260806 02:53:16.698267 11993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.188@KRBTEST.COM" created.
I20260806 02:53:16.712520 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.700345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.24.130.190:39297)
0806 02:53:16.700470 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.700474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.700877 (+ 403us) server_negotiation.cc:407] Connection header received
0806 02:53:16.701332 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.701336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.701395 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.701500 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.702523 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.703119 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.704504 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.705040 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.705193 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.708196 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.708202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.708204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.708228 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.709910 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.710603 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.710607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.710608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.710748 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.711120 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.711125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.711126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.711361 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.711466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.712116 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.712297 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:53:16.713039 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.698526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59330)
0806 02:53:16.698862 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.698886 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.699009 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.701711 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.701716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.701911 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.702048 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.702056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.704060 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.704064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.704742 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.704753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.705469 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.705477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.705628 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.706270 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.706290 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.708077 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.710526 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.710530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.710533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.710855 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.711209 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.711212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.711214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.711311 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.711950 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.711954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.712067 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.712180 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.712919 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":565}
I20260806 02:53:16.713563 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.699398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.24.130.189:44399)
0806 02:53:16.699951 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.699973 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.700028 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:53:16.701138 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.701168 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.701457 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.701717 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.702205 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.703896 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.704864 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.705327 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.705430 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.708259 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.708269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.708286 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.708325 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.710397 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.710984 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.710991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.710995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.711086 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.711423 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.711427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.711429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.711814 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.711971 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.712300 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.712460 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":358,"thread_start_us":158,"threads_started":1}
I20260806 02:53:16.715139 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.700321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39297 (local address 127.0.0.1:49276)
0806 02:53:16.700729 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.700781 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.701127 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.701557 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.701566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.702076 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.702383 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.702394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.703243 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.703248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.704353 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.704365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.705222 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.705226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.705389 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.706095 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.706120 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.708015 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.710041 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.710052 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.710063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.710489 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.710857 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.710860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.710862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.711017 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.711526 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.711535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.711860 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.712050 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.712415 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":103,"spinlock_wait_cycles":16512,"thread_start_us":84,"threads_started":1}
I20260806 02:53:16.716332 11993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.716449 11884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.717541 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.717277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48746)
0806 02:53:16.717401 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.717462 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 02:53:16.718724 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.718396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48756)
0806 02:53:16.718565 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.718617 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260806 02:53:16.718984 11884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:40155: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:53:16.725490 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.716965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59346)
0806 02:53:16.717088 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.717103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.717199 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.717539 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.717541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.717747 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.717865 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.717872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.718817 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.718821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.719377 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.719385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.719872 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.719875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.719969 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.720499 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.720520 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.721772 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.723837 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.723843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.723846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.724047 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.724375 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.724378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.724380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.724460 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.724903 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.724906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.725003 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.725115 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.725289 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 02:53:16.725525 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.717016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.24.130.189:44399)
0806 02:53:16.717139 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.717143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.717156 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:53:16.717308 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.717314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.717358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.717424 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.717985 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.718691 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.719496 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.719755 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.719871 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.721906 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.721910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.721912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.721935 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.723716 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.724150 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.724155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.724157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.724273 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.724549 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.724554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.724555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.724752 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.724914 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.725158 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.725365 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":61}
I20260806 02:53:16.726419 11993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.727281 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.726973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48768)
0806 02:53:16.727091 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.727220 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:53:16.728011 11920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
I20260806 02:53:16.729027 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.728787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48780)
0806 02:53:16.728912 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.728973 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:53:16.729238 11993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:40155: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab.
Entry for principal kudu/127.24.130.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab.
I20260806 02:53:16.733634 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:40155
--webserver_interface=127.24.130.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab
--principal=kudu/127.24.130.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39297,127.24.130.189:44399,127.24.130.188:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:16.750173 11884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } attempt: 1
I20260806 02:53:16.751223 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.750895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48796)
0806 02:53:16.751057 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.751135 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:53:16.752091 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.751866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48808)
0806 02:53:16.751984 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.752035 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:53:16.752283 11884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:40155: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
I20260806 02:53:16.778566 11993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } attempt: 1
I20260806 02:53:16.779754 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.779377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48818)
0806 02:53:16.779591 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.779672 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 02:53:16.780575 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.780361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48826)
0806 02:53:16.780470 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.780518 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:53:16.780753 11993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:40155: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
I20260806 02:53:16.827469 11884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } attempt: 2
I20260806 02:53:16.828537 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.828194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48830)
0806 02:53:16.828385 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.828456 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 02:53:16.829453 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.829160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48838)
0806 02:53:16.829299 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.829395 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 02:53:16.829602 11884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:40155: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
I20260806 02:53:16.848950 11993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } attempt: 2
I20260806 02:53:16.849901 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.849548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48848)
0806 02:53:16.849703 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.849789 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:53:16.850809 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.850512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48856)
0806 02:53:16.850623 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.850706 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:53:16.851037 11993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:40155: Network error: Client connection negotiation failed: client connection to 127.24.130.188:40155: connect: Connection refused (error 111)
W20260806 02:53:16.857366 12001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:16.857654 12001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:16.857740 12001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:16.862082 12001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:16.862170 12001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:16.862193 12001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:16.862249 12001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:16.862288 12001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:16.868041 12001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39297,127.24.130.189:44399,127.24.130.188:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab
--principal=kudu/127.24.130.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.130.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:16.869180 12001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:16.870267 12001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:16.879405 12015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:16.879434 12009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:16.879437 12010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:16.879488 12001 server_base.cc:1034] running on GCE node
I20260806 02:53:16.880245 12001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:16.881464 12001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:16.882714 12001 hybrid_clock.cc:648] HybridClock initialized: now 1785984796882693 us; error 45 us; skew 500 ppm
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:16.885618 12001 init.cc:377] Logged in from keytab as kudu/127.24.130.188@KRBTEST.COM (short username kudu)
I20260806 02:53:16.886591 12001 webserver.cc:460] Webserver started at http://127.24.130.188:36447/ using document root <none> and password file <none>
I20260806 02:53:16.887200 12001 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:16.887301 12001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:16.887553 12001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:16.889716 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data/instance:
uuid: "2015820513e843f8b6f23035cbfcbe12"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "e755ad0bc5d58e475aa1a78f563578cd"
server_key_iv: "0fb094c1e7141a15e6a7856e431b2c60"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.890338 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal/instance:
uuid: "2015820513e843f8b6f23035cbfcbe12"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "e755ad0bc5d58e475aa1a78f563578cd"
server_key_iv: "0fb094c1e7141a15e6a7856e431b2c60"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.894269 12001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:53:16.896883 12026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:16.898087 12001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:53:16.898236 12001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal
uuid: "2015820513e843f8b6f23035cbfcbe12"
format_stamp: "Formatted at 2026-08-06 02:53:16 on dist-test-slave-3kk6"
server_key: "e755ad0bc5d58e475aa1a78f563578cd"
server_key_iv: "0fb094c1e7141a15e6a7856e431b2c60"
server_key_version: "encryptionkey@0"
I20260806 02:53:16.898363 12001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:16.914985 12001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:16.918251 12001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:16.918497 12001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:16.926280 12001 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.188:40155
I20260806 02:53:16.926613 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.188:40155 every 8 connection(s)
I20260806 02:53:16.927444 12001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
I20260806 02:53:16.928787 11884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } attempt: 3
I20260806 02:53:16.930913 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:16.932336 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 12001
I20260806 02:53:16.932492 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984624291587-25098-0/minicluster-data/master-2/wal/instance
I20260806 02:53:16.932803 25098 external_mini_cluster.cc:1428] Setting key cd7f8721efffa46d708b8da57c1f52e7
I20260806 02:53:16.935745 12098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:53:16.942257 11993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } attempt: 3
I20260806 02:53:16.944691 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.929311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48870)
0806 02:53:16.929463 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.929483 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.929593 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.932198 (+ 2605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.932205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.932418 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.932546 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.932555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.934390 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.934395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.935262 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.935280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.936283 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.936287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.936377 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.937022 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.937038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.938741 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.942433 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.942439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.942441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.942751 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.943210 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.943216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.943219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.943321 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.944146 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.944150 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.944259 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.944349 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.944506 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:16.946229 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.933797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.24.130.188:40155)
0806 02:53:16.934146 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.934156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.934170 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:53:16.934206 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.934208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.934251 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.934346 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.935553 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.936289 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.937327 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.937689 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.937799 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.940802 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.940808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.940814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.940848 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.942695 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.943233 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.943237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.943239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.943315 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.943675 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.943678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.943680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.944011 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.944212 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.944741 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.945086 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":164,"threads_started":1}
I20260806 02:53:16.946228 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.930123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.24.130.188:40155)
0806 02:53:16.930699 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.930724 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.930764 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:53:16.931483 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.931501 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.931854 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.932225 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.932714 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.934240 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.935417 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.936144 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.936302 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.938967 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.938976 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.938995 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.939031 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.942282 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.942904 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.942911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.942919 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.943075 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.943434 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.943437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.943440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.944011 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.944996 (+ 985us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.945076 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.945209 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":325,"thread_start_us":154,"threads_started":1}
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:53:16.950184 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.938690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.24.130.190:39297)
0806 02:53:16.938849 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.938856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.938868 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:53:16.938906 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.938909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.938952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.939037 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.939956 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.940780 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.941919 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.942158 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.942296 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.946073 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.946077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.946080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.946112 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.947929 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.948539 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.948544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.948547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.948652 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.949028 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.949033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.949035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.949345 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.949456 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.949887 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.950027 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:53:16.951192 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.937762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39297 (local address 127.0.0.1:49282)
0806 02:53:16.938455 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.938513 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.938631 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.939112 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.939121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.939673 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.939811 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.939818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.940905 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.940909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.941797 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.941811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.942601 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.942607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.942766 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.943501 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.943521 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.945904 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.948062 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.948070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.948076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.948410 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.948781 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.948784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.948786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.948889 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.949519 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.949525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.949650 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.950427 (+ 777us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.950638 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":134,"threads_started":1}
I20260806 02:53:16.952327 11884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: Bootstrap starting.
I20260806 02:53:16.953502 12098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.954705 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.942826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48890)
0806 02:53:16.943132 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.943148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.943248 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.944050 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.944055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.944256 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.944399 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.944409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.946052 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.946056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.947437 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.947450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.948076 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.948080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.948170 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.948661 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.948681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.950254 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.952485 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.952489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.952491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.952764 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.953101 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.953104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.953106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.953201 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.954116 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.954122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.954258 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.954348 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.954503 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260806 02:53:16.955482 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.943459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.24.130.188:40155)
0806 02:53:16.943779 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.943788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.943803 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:53:16.943838 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.943841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.943882 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.944541 (+ 659us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.944655 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.945905 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.947578 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.947933 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.948394 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.950510 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.950513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.950516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.950541 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.952340 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.952896 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.952899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.952901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.952991 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.953333 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.953336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.953338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.953987 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.955115 (+ 1128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.955192 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.955325 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260806 02:53:16.956290 11884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: Neither blocks nor log segments found. Creating new log.
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:16.957221 11884 log.cc:824] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:16.959649 11884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: No bootstrap required, opened a new log
I20260806 02:53:16.959779 11993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: Bootstrap starting.
I20260806 02:53:16.960986 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.949866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.24.130.190:39297)
0806 02:53:16.950353 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.950356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.950372 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:53:16.950579 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.950583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.950637 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.950716 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.951809 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.952327 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.953387 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.953724 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.953825 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.956965 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.956969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.956972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.956998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.958823 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.959341 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.959345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.959347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.959441 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.959823 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.959829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.959833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.960089 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.960242 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.960670 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.960803 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":99,"threads_started":1}
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:53:16.962512 11993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:16.963441 11993 log.cc:824] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:16.964295 11884 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:16.964658 11884 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:16.964773 11884 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6d7b4f0c854fbd80b75aa116626a20, State: Initialized, Role: FOLLOWER
I20260806 02:53:16.965389 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.954598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.24.130.189:44399)
0806 02:53:16.954817 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.954824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.954844 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:53:16.955022 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.955025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.955082 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.955146 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.955776 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.956557 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.958593 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.958991 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.959075 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.961539 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.961544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.961547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.961575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.963354 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.963903 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.963909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.963911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.964004 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.964368 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.964372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.964373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.964664 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.964741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.965115 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.965236 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:53:16.965677 11884 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:16.965808 11993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: No bootstrap required, opened a new log
I20260806 02:53:16.965921 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.954023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:44399 (local address 127.0.0.1:59350)
0806 02:53:16.954784 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.954801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.954888 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.955311 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.955315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.955517 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.955623 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.955631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.956900 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.956905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.958435 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.958448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.959242 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.959246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.959348 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.959862 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.959882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.961357 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.963509 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.963512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.963514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.963786 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.964143 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.964148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.964152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.964255 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.964821 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.964824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.964922 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.965592 (+ 670us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.965758 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680}
I20260806 02:53:16.966791 12098 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:53:16.967715 11884 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:53:16.970156 11993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:16.970383 12110 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:16.970537 12110 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:16.971529 11993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:16.971661 11993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d00cc7e07c4285853e1f103c1bcb71, State: Initialized, Role: FOLLOWER
W20260806 02:53:16.974216 12128 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:53:16.974357 12128 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:53:16.975422 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.962868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.24.130.189:44399)
0806 02:53:16.963568 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.963578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.963592 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:53:16.963628 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.963631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.963678 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.964599 (+ 921us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.964702 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.965490 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.966577 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.966974 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.967529 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.970053 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.970056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.970058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.970080 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.973292 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.973845 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.973849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.973850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.973956 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.974302 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.974306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.974307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.974547 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.974801 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.975020 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.975166 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":138,"threads_started":1}
Aug 06 02:53:16 dist-test-slave-3kk6 krb5kdc[11768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984796, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:53:16.977480 11993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:16.973474 11787 master_runner.cc:186] Error getting master registration for 127.24.130.189:44399: OK, instance_id { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" instance_seqno: 1785984796687591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:53:16.977989 11787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:53:16.979092 11993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:16.979321 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:53:16.980881 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.970724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:40155 (local address 127.0.0.1:48898)
0806 02:53:16.970956 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.970978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.971090 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.971476 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.971480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.971689 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.971792 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.971799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.973302 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.973305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.974091 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.974102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.974794 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.974797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.974880 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.975347 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.975367 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.977104 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:16.979260 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.979263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.979265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.979520 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.979777 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:16.979781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:16.979785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:16.979867 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:16.980305 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:16.980307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:16.980390 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:53:16.980504 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:53:16.980654 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260806 02:53:16.981333 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.970854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.24.130.188:40155)
0806 02:53:16.971244 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.971249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.971260 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:53:16.971291 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.971294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.971331 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.971411 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.971905 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.972648 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.974228 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.974561 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.974685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.977244 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.977248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.977250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.977280 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.979148 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.979622 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.979625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.979626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.979695 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.979958 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.979961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.979963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.980166 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.980270 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.981081 (+ 811us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.981198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260806 02:53:16.984468 12098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12: Bootstrap starting.
W20260806 02:53:16.985641 12152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:53:16.985723 12152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:53:16.987341 12098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:16.988219 12098 log.cc:824] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:16.987317 11897 master_runner.cc:186] Error getting master registration for 127.24.130.188:40155: OK, instance_id { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" instance_seqno: 1785984796924168 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:53:16.991504 11897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:53:16.994922 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:16.995110 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:16.998142 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.986891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.24.130.188:40155)
0806 02:53:16.987016 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.987020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.988405 (+ 1385us) server_negotiation.cc:407] Connection header received
0806 02:53:16.988596 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.988599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.988647 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.988698 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.989787 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.990544 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.991902 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.992261 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.992363 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.994297 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.994301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.994304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.994331 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.996131 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.996712 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.996717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.996720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.996807 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.997128 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.997131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.997133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.997367 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:16.997484 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:16.997859 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 02:53:16.997982 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:53:17.001089 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.987408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.24.130.189:44399)
0806 02:53:16.991119 (+ 3711us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.991125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.991140 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:53:16.991182 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.991188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.991236 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.991327 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.992523 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.993317 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.994353 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.994753 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.994842 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.996797 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.996801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.996804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.996832 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.998598 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:16.999682 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:16.999688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:16.999693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:16.999792 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:17.000146 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:17.000152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:17.000156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:17.000387 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:17.000531 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:17.000807 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:53:17.000931 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3615}
I20260806 02:53:17.001827 12098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12: No bootstrap required, opened a new log
I20260806 02:53:17.004277 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.990482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.24.130.190:39297)
0806 02:53:16.990608 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.990615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.990629 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:53:16.990725 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.990727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.990777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.990872 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.991828 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.992598 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.994063 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.994418 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.994488 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.996979 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:16.996983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:16.996986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:16.997018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:16.999232 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:17.002814 (+ 3582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:17.002818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:17.002820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:17.002925 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:17.003262 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:17.003266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:17.003268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:17.003497 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:17.003626 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:17.003992 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 02:53:17.004129 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:53:17.007046 12098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:17.007299 12098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:17.007395 12098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2015820513e843f8b6f23035cbfcbe12, State: Initialized, Role: FOLLOWER
I20260806 02:53:17.007936 12098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:17.008492 12160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.008685 12160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:17.009364 12098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:53:17.015136 12175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:53:17.015365 12175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:53:17.021332 12001 master_runner.cc:263] No leader master found from master 2015820513e843f8b6f23035cbfcbe12
I20260806 02:53:17.021441 12001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:53:17.029706 12160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:17.029914 12160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:17.031016 12160 leader_election.cc:288] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc6d7b4f0c854fbd80b75aa116626a20 (127.24.130.190:39297), 42d00cc7e07c4285853e1f103c1bcb71 (127.24.130.189:44399)
I20260806 02:53:17.031956 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2015820513e843f8b6f23035cbfcbe12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d00cc7e07c4285853e1f103c1bcb71" is_pre_election: true
I20260806 02:53:17.032301 11960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2015820513e843f8b6f23035cbfcbe12 in term 0.
I20260806 02:53:17.032796 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2015820513e843f8b6f23035cbfcbe12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" is_pre_election: true
I20260806 02:53:17.033105 11851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2015820513e843f8b6f23035cbfcbe12 in term 0.
I20260806 02:53:17.033553 12029 leader_election.cc:302] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2015820513e843f8b6f23035cbfcbe12, 42d00cc7e07c4285853e1f103c1bcb71; no voters:
I20260806 02:53:17.034368 12160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:53:17.034466 12160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:17.034510 12160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:17.035979 12160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:17.036414 12160 leader_election.cc:288] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [CANDIDATE]: Term 1 election: Requested vote from peers bc6d7b4f0c854fbd80b75aa116626a20 (127.24.130.190:39297), 42d00cc7e07c4285853e1f103c1bcb71 (127.24.130.189:44399)
I20260806 02:53:17.036810 11851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2015820513e843f8b6f23035cbfcbe12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6d7b4f0c854fbd80b75aa116626a20"
I20260806 02:53:17.036936 11851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:17.039194 11851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2015820513e843f8b6f23035cbfcbe12 in term 1.
I20260806 02:53:17.039477 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2015820513e843f8b6f23035cbfcbe12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d00cc7e07c4285853e1f103c1bcb71"
I20260806 02:53:17.039610 11960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:17.039597 12030 leader_election.cc:302] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2015820513e843f8b6f23035cbfcbe12, bc6d7b4f0c854fbd80b75aa116626a20; no voters:
I20260806 02:53:17.039800 12160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:17.040014 12160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [term 1 LEADER]: Becoming Leader. State: Replica: 2015820513e843f8b6f23035cbfcbe12, State: Running, Role: LEADER
I20260806 02:53:17.040431 12160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } }
I20260806 02:53:17.040935 11960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2015820513e843f8b6f23035cbfcbe12 in term 1.
I20260806 02:53:17.044341 12160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2015820513e843f8b6f23035cbfcbe12. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.044576 12160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:17.045023 12181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:17.046029 12181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:17.051888 11960 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [term 1 FOLLOWER]: Refusing update from remote peer 2015820513e843f8b6f23035cbfcbe12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:17.052837 12160 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:17.057695 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2015820513e843f8b6f23035cbfcbe12. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.057842 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:17.059880 11851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [term 1 FOLLOWER]: Refusing update from remote peer 2015820513e843f8b6f23035cbfcbe12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:17.060612 12160 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:17.065021 12110 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2015820513e843f8b6f23035cbfcbe12. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.065166 12110 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:17.067418 12160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.067546 12160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:17.068677 12183 mvcc.cc:205] Tried to move back new op lower bound from 7315393728717000704 to 7315393728688959488. Current Snapshot: MvccSnapshot[applied={T|T < 7315393728717000704}]
I20260806 02:53:17.068931 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.069063 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:17.070710 12182 mvcc.cc:205] Tried to move back new op lower bound from 7315393728717000704 to 7315393728688959488. Current Snapshot: MvccSnapshot[applied={T|T < 7315393728717000704}]
I20260806 02:53:17.072059 12178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.072211 12178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:17.073031 12110 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2015820513e843f8b6f23035cbfcbe12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6d7b4f0c854fbd80b75aa116626a20" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39297 } } peers { permanent_uuid: "42d00cc7e07c4285853e1f103c1bcb71" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 44399 } } peers { permanent_uuid: "2015820513e843f8b6f23035cbfcbe12" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 40155 } } }
I20260806 02:53:17.073158 12110 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:53:17.074338 12185 mvcc.cc:205] Tried to move back new op lower bound from 7315393728717000704 to 7315393728688959488. Current Snapshot: MvccSnapshot[applied={T|T < 7315393728717000704}]
I20260806 02:53:17.076241 12181 catalog_manager.cc:1347] Generated new cluster ID: b73851100eea4d7e807c05f8197c473d
I20260806 02:53:17.076304 12181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:17.098887 12181 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:17.100350 12181 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:17.120033 12181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2015820513e843f8b6f23035cbfcbe12: Generated new TSK 0
I20260806 02:53:17.120985 12181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:53:17.976015 12128 catalog_manager.cc:1259] Loaded cluster ID: b73851100eea4d7e807c05f8197c473d
I20260806 02:53:17.976122 12128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: loading cluster ID for follower catalog manager: success
I20260806 02:53:17.977814 12128 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: acquiring CA information for follower catalog manager: success
I20260806 02:53:17.979139 12128 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc6d7b4f0c854fbd80b75aa116626a20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:53:17.987102 12152 catalog_manager.cc:1259] Loaded cluster ID: b73851100eea4d7e807c05f8197c473d
I20260806 02:53:17.987191 12152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: loading cluster ID for follower catalog manager: success
I20260806 02:53:17.988808 12152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: acquiring CA information for follower catalog manager: success
I20260806 02:53:17.989964 12152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42d00cc7e07c4285853e1f103c1bcb71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:53:18.201558 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 11787
I20260806 02:53:18.213454 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 11897
I20260806 02:53:18.225857 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 12001
2026-08-06T02:53:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3325 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:18 dist-test-slave-3kk6 krb5kdc[12202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:18 dist-test-slave-3kk6 krb5kdc[12202](info): set up 2 sockets
Aug 06 02:53:18 dist-test-slave-3kk6 krb5kdc[12202](info): commencing operation
krb5kdc: starting...
W20260806 02:53:19.460325 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:53:19 dist-test-slave-3kk6 krb5kdc[12202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:53:19.604349 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:37607
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:37607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:19.725705 12223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:19.726099 12223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:19.726177 12223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:19.730417 12223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:19.730554 12223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:19.730592 12223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:19.730629 12223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:19.730720 12223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:19.736682 12223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:37607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:19.738045 12223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:19.739252 12223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:19.748768 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:19.748833 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:19.748768 12232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:19.748974 12223 server_base.cc:1034] running on GCE node
I20260806 02:53:19.749644 12223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:19.750999 12223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:19.752188 12223 hybrid_clock.cc:648] HybridClock initialized: now 1785984799752163 us; error 46 us; skew 500 ppm
Aug 06 02:53:19 dist-test-slave-3kk6 krb5kdc[12202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984799, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:19.755271 12223 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:19.756424 12223 webserver.cc:460] Webserver started at http://127.24.130.190:43819/ using document root <none> and password file <none>
I20260806 02:53:19.757087 12223 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:19.757159 12223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:19.757409 12223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:19.759860 12223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "774baadeefe545568e8e8e41fcc0fb69"
format_stamp: "Formatted at 2026-08-06 02:53:19 on dist-test-slave-3kk6"
server_key: "148377bed640ee1180f2789cf6128cac"
server_key_iv: "9be5a63a3d982e10c2a48058ec46c60b"
server_key_version: "encryptionkey@0"
I20260806 02:53:19.760567 12223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "774baadeefe545568e8e8e41fcc0fb69"
format_stamp: "Formatted at 2026-08-06 02:53:19 on dist-test-slave-3kk6"
server_key: "148377bed640ee1180f2789cf6128cac"
server_key_iv: "9be5a63a3d982e10c2a48058ec46c60b"
server_key_version: "encryptionkey@0"
I20260806 02:53:19.765069 12223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:53:19.768054 12243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:19.769479 12223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:53:19.769646 12223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "774baadeefe545568e8e8e41fcc0fb69"
format_stamp: "Formatted at 2026-08-06 02:53:19 on dist-test-slave-3kk6"
server_key: "148377bed640ee1180f2789cf6128cac"
server_key_iv: "9be5a63a3d982e10c2a48058ec46c60b"
server_key_version: "encryptionkey@0"
I20260806 02:53:19.769838 12223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:19.786352 12223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:19.790146 12223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:19.790354 12223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:19.799024 12223 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:37607
I20260806 02:53:19.799041 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:37607 every 8 connection(s)
I20260806 02:53:19.800195 12223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:19.803579 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:19.803808 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 12223
I20260806 02:53:19.803982 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:53:19.804322 25098 external_mini_cluster.cc:1428] Setting key 3ea95d94fc6ac43baad852b6dc38a686
I20260806 02:53:19.811537 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69: Bootstrap starting.
I20260806 02:53:19.814694 12316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69: Neither blocks nor log segments found. Creating new log.
Aug 06 02:53:19 dist-test-slave-3kk6 krb5kdc[12202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:19.815732 12316 log.cc:824] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:19.818101 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69: No bootstrap required, opened a new log
I20260806 02:53:19.820888 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:19.806012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.24.130.190:37607)
0806 02:53:19.806542 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:19.806559 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:19.806600 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:53:19.807392 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:19.807412 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:19.807782 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:19.808099 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:19.809210 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.810272 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.811586 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.812172 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.812342 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.815561 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:19.815567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:19.815582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:19.815616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:19.817544 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:19.818155 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:19.818161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:19.818166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:19.818406 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:19.818786 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:19.818789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:19.818791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:19.819223 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:19.819403 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:19.819720 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 02:53:19.820076 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":201,"threads_started":1}
I20260806 02:53:19.822747 12316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774baadeefe545568e8e8e41fcc0fb69" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37607 } }
I20260806 02:53:19.823079 12316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:19.823191 12316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774baadeefe545568e8e8e41fcc0fb69, State: Initialized, Role: FOLLOWER
I20260806 02:53:19.823726 12316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774baadeefe545568e8e8e41fcc0fb69" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37607 } }
I20260806 02:53:19.823899 12316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:19.823993 12316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:19.824117 12316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:19.824426 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:53:19.825285 12316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774baadeefe545568e8e8e41fcc0fb69" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37607 } }
I20260806 02:53:19.825713 12316 leader_election.cc:302] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 774baadeefe545568e8e8e41fcc0fb69; no voters:
I20260806 02:53:19.826148 12316 leader_election.cc:288] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:19.826294 12321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:19.826534 12321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [term 1 LEADER]: Becoming Leader. State: Replica: 774baadeefe545568e8e8e41fcc0fb69, State: Running, Role: LEADER
I20260806 02:53:19.826941 12321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774baadeefe545568e8e8e41fcc0fb69" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37607 } }
I20260806 02:53:19.827594 12316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:19.828814 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "774baadeefe545568e8e8e41fcc0fb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774baadeefe545568e8e8e41fcc0fb69" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37607 } } }
I20260806 02:53:19.828842 12323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774baadeefe545568e8e8e41fcc0fb69. Latest consensus state: current_term: 1 leader_uuid: "774baadeefe545568e8e8e41fcc0fb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774baadeefe545568e8e8e41fcc0fb69" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37607 } } }
I20260806 02:53:19.828999 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:19.829000 12323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:19.829412 12333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:19.832397 12333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:19.839222 12333 catalog_manager.cc:1347] Generated new cluster ID: a50618fa4ff94af6a98c0c6ad1a4eb59
I20260806 02:53:19.839340 12333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:19.857633 12333 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:19.859143 12333 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:19.865909 12333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774baadeefe545568e8e8e41fcc0fb69: Generated new TSK 0
I20260806 02:53:19.866918 12333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:53:19.881220 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 12223
2026-08-06T02:53:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1659 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:19 dist-test-slave-3kk6 krb5kdc[12352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:19 dist-test-slave-3kk6 krb5kdc[12352](info): set up 2 sockets
Aug 06 02:53:19 dist-test-slave-3kk6 krb5kdc[12352](info): commencing operation
krb5kdc: starting...
W20260806 02:53:21.139091 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:53:21 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:53:21.279575 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38563
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:21.402293 12366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:21.402704 12366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:21.402817 12366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:21.407075 12366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:21.407183 12366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:21.407218 12366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:21.407251 12366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:21.407310 12366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:21.413105 12366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:21.414331 12366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:21.415510 12366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:21.424635 12376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:21.424630 12377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:21.424687 12366 server_base.cc:1034] running on GCE node
W20260806 02:53:21.424628 12380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:21.425725 12366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:21.427035 12366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:21.428236 12366 hybrid_clock.cc:648] HybridClock initialized: now 1785984801428211 us; error 45 us; skew 500 ppm
Aug 06 02:53:21 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:21.431392 12366 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:21.432428 12366 webserver.cc:460] Webserver started at http://127.24.130.190:36925/ using document root <none> and password file <none>
I20260806 02:53:21.433043 12366 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:21.433144 12366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:21.433403 12366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:21.435755 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "ca1b588a06cf432798fd24470d3de815"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "e13bb2374e4db6753d206f817c4798a1"
server_key_iv: "5f30b20c9487e5891327bb6aee9b854c"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.436419 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "ca1b588a06cf432798fd24470d3de815"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "e13bb2374e4db6753d206f817c4798a1"
server_key_iv: "5f30b20c9487e5891327bb6aee9b854c"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.440948 12366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:53:21.443935 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:21.445389 12366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:21.445526 12366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "ca1b588a06cf432798fd24470d3de815"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "e13bb2374e4db6753d206f817c4798a1"
server_key_iv: "5f30b20c9487e5891327bb6aee9b854c"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.445642 12366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:21.470327 12366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:21.474056 12366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:21.474277 12366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:21.482283 12366 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:38563
I20260806 02:53:21.482733 12459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:38563 every 8 connection(s)
I20260806 02:53:21.483505 12366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:21.486526 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:21.489080 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 12366
I20260806 02:53:21.489252 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:53:21.489552 25098 external_mini_cluster.cc:1428] Setting key cb11981d64679c5f170a45ab566db28b
I20260806 02:53:21.493465 12460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap starting.
I20260806 02:53:21.496316 12460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:21.497231 12460 log.cc:824] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:21.499612 12460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: No bootstrap required, opened a new log
Aug 06 02:53:21 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:21.504427 12460 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:21.504763 12460 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:21.504899 12460 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1b588a06cf432798fd24470d3de815, State: Initialized, Role: FOLLOWER
I20260806 02:53:21.505436 12460 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:21.505573 12460 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:21.505663 12460 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:21.505832 12460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:21.506158 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.491221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.24.130.190:38563)
0806 02:53:21.491715 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.491737 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.491787 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:53:21.492661 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.492685 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.493113 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.493505 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.494326 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.495899 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.496843 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.497599 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.497771 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.500871 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:21.500876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:21.500891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:21.500922 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:21.502869 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:21.503416 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:21.503421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:21.503426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:21.503546 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:21.503852 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:21.503854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:21.503855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:21.504281 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:21.504464 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:21.504813 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:53:21.505186 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":129,"threads_started":1}
I20260806 02:53:21.507012 12460 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:21.507419 12460 leader_election.cc:302] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1b588a06cf432798fd24470d3de815; no voters:
I20260806 02:53:21.507802 12460 leader_election.cc:288] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:21.507916 12467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:21.508183 12467 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 LEADER]: Becoming Leader. State: Replica: ca1b588a06cf432798fd24470d3de815, State: Running, Role: LEADER
I20260806 02:53:21.508876 12467 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:21.509188 12460 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:21.510506 12469 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1b588a06cf432798fd24470d3de815. Latest consensus state: current_term: 1 leader_uuid: "ca1b588a06cf432798fd24470d3de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } } }
I20260806 02:53:21.510635 12469 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:21.510815 12468 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca1b588a06cf432798fd24470d3de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } } }
I20260806 02:53:21.511243 12468 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:21.511348 12484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:21.515197 12484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:21.521057 12484 catalog_manager.cc:1347] Generated new cluster ID: 50850d2b26a54e8d89881c7857e3d9c3
I20260806 02:53:21.521147 12484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:21.535153 12484 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:21.536265 12484 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:21.542119 12484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Generated new TSK 0
I20260806 02:53:21.543020 12484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:53:21.577934 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38563
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:21.712843 12493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:21.713223 12493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:21.713366 12493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:21.718379 12493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:21.718508 12493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:21.718621 12493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:53:21.725178 12493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:21.726553 12493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:21.727849 12493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:21.739372 12501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:21.739372 12500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:21.739418 12503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:21.739722 12493 server_base.cc:1034] running on GCE node
I20260806 02:53:21.740413 12493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:21.741220 12493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:21.742455 12493 hybrid_clock.cc:648] HybridClock initialized: now 1785984801742440 us; error 31 us; skew 500 ppm
Aug 06 02:53:21 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:21.745785 12493 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:53:21.747068 12493 webserver.cc:460] Webserver started at http://127.24.130.129:45661/ using document root <none> and password file <none>
I20260806 02:53:21.747745 12493 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:21.747855 12493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:21.748135 12493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:21.751171 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "b0ed219a3c264d8f860d6e82b3de0fca"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "cdb3647652cad051536b65a094d65c16"
server_key_iv: "fb76aa30db5cecf44f7ad86fed93b11b"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.751916 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "b0ed219a3c264d8f860d6e82b3de0fca"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "cdb3647652cad051536b65a094d65c16"
server_key_iv: "fb76aa30db5cecf44f7ad86fed93b11b"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.756470 12493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:53:21.759557 12512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:21.761204 12493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:21.761404 12493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "b0ed219a3c264d8f860d6e82b3de0fca"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "cdb3647652cad051536b65a094d65c16"
server_key_iv: "fb76aa30db5cecf44f7ad86fed93b11b"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.761583 12493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:21.775643 12493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:21.779080 12493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:21.779320 12493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:21.779999 12493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:21.780946 12493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:21.781023 12493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:21.781100 12493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:21.781143 12493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:21.791379 12493 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:43247
I20260806 02:53:21.791399 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:43247 every 8 connection(s)
I20260806 02:53:21.792418 12493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:21.797747 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 12493
I20260806 02:53:21.797922 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:21.798264 25098 external_mini_cluster.cc:1428] Setting key e7994e5c78e0fa7b79414f8abefc763c
Aug 06 02:53:21 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:21.809554 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.794507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47451 (local address 127.24.130.190:38563)
0806 02:53:21.794709 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.794713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.795700 (+ 987us) server_negotiation.cc:407] Connection header received
0806 02:53:21.797149 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.797152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.797210 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.797298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.799148 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.799769 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.801136 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.801550 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.801677 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.805042 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:21.805046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:21.805048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:21.805078 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:21.806925 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:21.807616 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:21.807620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:21.807621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:21.807712 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:21.808139 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:21.808143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:21.808147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:21.808376 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:21.808500 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:21.809199 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 02:53:21.809360 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:53:21.810743 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.794992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:47451)
0806 02:53:21.795518 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:21.795569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:21.796904 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:21.797548 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:21.797561 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:21.798184 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:21.798918 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.798937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.799970 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.799975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.800984 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.800995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.801713 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.801716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.801884 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.802690 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:21.802722 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:21.804797 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:21.807090 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:21.807098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:21.807128 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:21.807469 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:21.807832 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:21.807835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:21.807838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:21.808014 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:21.808500 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:21.808509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:21.808822 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 02:53:21.809134 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 02:53:21.809587 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":119,"threads_started":1}
I20260806 02:53:21.812309 12678 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38563
I20260806 02:53:21.812629 12678 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:53:21.813349 12678 heartbeater.cc:511] Master 127.24.130.190:38563 requested a full tablet report, sending...
I20260806 02:53:21.815240 12411 ts_manager.cc:194] Registered new tserver with Master: b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247)
I20260806 02:53:21.816807 12411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:47451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:53:21.839434 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38563
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:21.960734 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:21.961052 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:21.961102 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:21.965299 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:21.965394 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:21.965457 12684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:53:21.971140 12684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:21.972321 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:21.973397 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:21.983800 12692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:21.983774 12690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:21.983774 12689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:21.984138 12684 server_base.cc:1034] running on GCE node
I20260806 02:53:21.984638 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:21.985351 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:21.986532 12684 hybrid_clock.cc:648] HybridClock initialized: now 1785984801986474 us; error 76 us; skew 500 ppm
Aug 06 02:53:21 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:21.989763 12684 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:53:21.990835 12684 webserver.cc:460] Webserver started at http://127.24.130.130:38941/ using document root <none> and password file <none>
I20260806 02:53:21.991462 12684 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:21.991567 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:21.991838 12684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:21.994160 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "ee2099cd278565a948ef1f7b13b2f54c"
server_key_iv: "98ab7942fd859d7c357401cdbf830677"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.994974 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "ee2099cd278565a948ef1f7b13b2f54c"
server_key_iv: "98ab7942fd859d7c357401cdbf830677"
server_key_version: "encryptionkey@0"
I20260806 02:53:21.999508 12684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:53:22.002218 12700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.003666 12684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:53:22.003832 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "ee2099cd278565a948ef1f7b13b2f54c"
server_key_iv: "98ab7942fd859d7c357401cdbf830677"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.003975 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:22.024991 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:22.028877 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:22.029124 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:22.029804 12684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:22.030813 12684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:22.030887 12684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.030964 12684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:22.031009 12684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.040601 12684 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:40813
I20260806 02:53:22.040685 12854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:40813 every 8 connection(s)
I20260806 02:53:22.041685 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:22.048834 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 12684
I20260806 02:53:22.048966 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:22.049326 25098 external_mini_cluster.cc:1428] Setting key c40ab3e70daf4f8362c535513998df66
Aug 06 02:53:22 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984801, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:22.058605 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.043931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:55521 (local address 127.24.130.190:38563)
0806 02:53:22.044089 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.044093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.045017 (+ 924us) server_negotiation.cc:407] Connection header received
0806 02:53:22.046481 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.046485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.046550 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.046661 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:22.048007 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.048616 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.050213 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.050588 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.050737 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.054161 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:22.054165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:22.054167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:22.054205 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:22.055872 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:22.056676 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:22.056679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:22.056680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:22.056766 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:22.057185 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:22.057189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:22.057193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:22.057423 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:22.057542 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.058334 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.058467 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":60}
I20260806 02:53:22.060110 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.044350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:55521)
0806 02:53:22.044846 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.044895 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.046230 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.046901 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.046908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.047322 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:22.047840 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.047855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.048783 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.048786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.050040 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.050052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.050819 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.050824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.051080 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.051826 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:22.051854 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:22.053901 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:22.056023 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:22.056033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:22.056058 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:22.056555 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:22.056884 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:22.056888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:22.056890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:22.057070 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:22.057546 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:22.057557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:22.057988 (+ 431us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.058273 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.058779 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260806 02:53:22.061666 12856 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38563
I20260806 02:53:22.061954 12856 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:53:22.062565 12856 heartbeater.cc:511] Master 127.24.130.190:38563 requested a full tablet report, sending...
I20260806 02:53:22.063802 12411 ts_manager.cc:194] Registered new tserver with Master: 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813)
I20260806 02:53:22.064379 12411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:55521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:53:22.083432 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38563
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:22.220569 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:22.220939 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:22.220991 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:22.225829 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:22.225929 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:22.225993 12866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:53:22.231855 12866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:22.233044 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:22.234233 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:22.244851 12877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:22.244874 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:22.245085 12866 server_base.cc:1034] running on GCE node
W20260806 02:53:22.244890 12879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:22.245632 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:22.246410 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:22.247622 12866 hybrid_clock.cc:648] HybridClock initialized: now 1785984802247611 us; error 72 us; skew 500 ppm
Aug 06 02:53:22 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984802, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:22.251052 12866 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:53:22.252314 12866 webserver.cc:460] Webserver started at http://127.24.130.131:36439/ using document root <none> and password file <none>
I20260806 02:53:22.252985 12866 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:22.253110 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:22.253369 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:22.255820 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "afca7f5211384dceab28249fd6b35176"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-3kk6"
server_key: "aa729f232fabb80e630008ab0979a816"
server_key_iv: "760623b62e85ba781de58373d93a81c8"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.256492 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "afca7f5211384dceab28249fd6b35176"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-3kk6"
server_key: "aa729f232fabb80e630008ab0979a816"
server_key_iv: "760623b62e85ba781de58373d93a81c8"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.260820 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:53:22.263772 12889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.265339 12866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:22.265506 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "afca7f5211384dceab28249fd6b35176"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-3kk6"
server_key: "aa729f232fabb80e630008ab0979a816"
server_key_iv: "760623b62e85ba781de58373d93a81c8"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.265650 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:22.296516 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:22.300365 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:22.300612 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:22.301306 12866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:22.302240 12866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:22.302318 12866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.302395 12866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:22.302449 12866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.313366 12866 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:32797
I20260806 02:53:22.313447 13043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:32797 every 8 connection(s)
I20260806 02:53:22.314754 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:22.323360 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 12866
I20260806 02:53:22.323529 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:22.323827 25098 external_mini_cluster.cc:1428] Setting key 8058b50905819224492a22812353823c
Aug 06 02:53:22 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984802, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:22.331133 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.317044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38113 (local address 127.24.130.190:38563)
0806 02:53:22.317237 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.317241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.318179 (+ 938us) server_negotiation.cc:407] Connection header received
0806 02:53:22.319768 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.319771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.319828 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.319922 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:22.321337 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.321914 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.323552 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.323911 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.324079 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.327494 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:22.327498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:22.327500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:22.327530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:22.328974 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:22.329517 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:22.329520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:22.329521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:22.329609 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:22.329944 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:22.329947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:22.329948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:22.330157 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:22.330241 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.330869 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.330995 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:53:22.332022 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.317437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:38113)
0806 02:53:22.317990 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.318043 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.319499 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.320121 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.320129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.320579 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:22.321126 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.321142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.322082 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.322085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.323306 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.323320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.324134 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.324139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.324389 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.325261 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:22.325320 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:22.327313 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:22.329096 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:22.329102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:22.329116 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:22.329409 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:22.329690 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:22.329692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:22.329694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:22.329847 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:22.330250 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:22.330255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:22.330528 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.330774 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.331093 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":144,"threads_started":1}
I20260806 02:53:22.333288 13044 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38563
I20260806 02:53:22.333559 13044 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:22.334113 13044 heartbeater.cc:511] Master 127.24.130.190:38563 requested a full tablet report, sending...
I20260806 02:53:22.335299 12411 ts_manager.cc:194] Registered new tserver with Master: afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797)
I20260806 02:53:22.335899 12411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:38113
I20260806 02:53:22.338415 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:53:22.349073 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.340701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.24.130.190:38563)
0806 02:53:22.340866 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.340872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.341028 (+ 156us) server_negotiation.cc:407] Connection header received
0806 02:53:22.341139 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.341142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.341185 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.341275 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:22.342059 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.342600 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.343507 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.343727 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.343820 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.345367 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:22.345370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:22.345373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:22.345400 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:22.347232 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:22.347693 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:22.347696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:22.347697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:22.347803 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:22.348130 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:22.348132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:22.348133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:22.348392 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:22.348501 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.348766 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.348904 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:53:22.352701 12411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:53:22.356014 12411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:53:22.372834 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.365584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43247 (local address 127.0.0.1:33328)
0806 02:53:22.366198 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.366262 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.366419 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.367629 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.367633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.367652 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:22.367863 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.367871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.369365 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.369371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.371108 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.371119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.372141 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.372146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.372339 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.372372 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.372504 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.372658 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":332,"spinlock_wait_cycles":9856,"thread_start_us":162,"threads_started":1}
I20260806 02:53:22.372884 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.367639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.24.130.131:32797)
0806 02:53:22.368100 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.368108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.368127 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:53:22.368185 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.368189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.368347 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.368503 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:22.368694 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.369747 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.371116 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.371980 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.372125 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.372271 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.372648 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.372749 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":163,"threads_started":1}
I20260806 02:53:22.373222 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.367169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32797 (local address 127.0.0.1:44230)
0806 02:53:22.367482 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.367498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.367577 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.368483 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.368487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.368505 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:22.368592 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.368599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.369901 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.369905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.370988 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.370998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.372141 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.372146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.372324 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.372373 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.372496 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.373106 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":486,"thread_start_us":127,"threads_started":1}
I20260806 02:53:22.373551 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.365718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.24.130.129:43247)
0806 02:53:22.366889 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.366897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.366927 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:53:22.367038 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.367046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.367287 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.367452 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:22.367984 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.369194 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.371243 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.372006 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.372125 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.372285 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.373332 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.373439 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":89,"threads_started":1}
I20260806 02:53:22.374112 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.366494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.24.130.130:40813)
0806 02:53:22.366844 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.366947 (+ 103us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.366975 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:53:22.367061 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.367066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.367250 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.367799 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:22.367983 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.369237 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.372374 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.373232 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.373365 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.373501 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.373913 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.374004 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":149,"threads_started":1}
I20260806 02:53:22.373816 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.365485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.0.0.1:55372)
0806 02:53:22.366198 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.366261 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.366419 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.367463 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.367469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.367496 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:22.367810 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.367820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.369383 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.369386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.371107 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.371119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.373362 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.373366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.373469 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.373491 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.373575 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.373661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":180,"spinlock_wait_cycles":1792,"thread_start_us":93,"threads_started":1}
I20260806 02:53:22.376972 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 7f139416452c46d4b7cabd2d6b9d7818 (DEFAULT_TABLE table=test-table [id=031b523687c447f98d70bd68c9e408ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:22.376989 12591 tablet_service.cc:1467] Processing CreateTablet for tablet 7f139416452c46d4b7cabd2d6b9d7818 (DEFAULT_TABLE table=test-table [id=031b523687c447f98d70bd68c9e408ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:22.377007 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 7f139416452c46d4b7cabd2d6b9d7818 (DEFAULT_TABLE table=test-table [id=031b523687c447f98d70bd68c9e408ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:22.378072 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f139416452c46d4b7cabd2d6b9d7818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:22.378093 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f139416452c46d4b7cabd2d6b9d7818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:22.378089 12591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f139416452c46d4b7cabd2d6b9d7818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:22.385186 13062 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Bootstrap starting.
I20260806 02:53:22.386328 13064 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Bootstrap starting.
I20260806 02:53:22.386904 13063 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap starting.
I20260806 02:53:22.388304 13064 tablet_bootstrap.cc:654] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:22.388365 13062 tablet_bootstrap.cc:654] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:22.389171 13063 tablet_bootstrap.cc:654] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:22.389371 13062 log.cc:824] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:22.389441 13064 log.cc:824] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:22.390003 13063 log.cc:824] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:22.391904 13064 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: No bootstrap required, opened a new log
I20260806 02:53:22.392021 13063 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: No bootstrap required, opened a new log
I20260806 02:53:22.392021 13062 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: No bootstrap required, opened a new log
I20260806 02:53:22.392166 13064 ts_tablet_manager.cc:1397] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:53:22.392223 13063 ts_tablet_manager.cc:1397] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:53:22.392223 13062 ts_tablet_manager.cc:1397] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:53:22.396561 13064 raft_consensus.cc:348] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.396561 13062 raft_consensus.cc:348] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.396940 13062 raft_consensus.cc:374] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:22.396944 13064 raft_consensus.cc:374] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:22.396801 13063 raft_consensus.cc:348] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.396996 13062 raft_consensus.cc:729] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20120fa7e3f54d3a9cf8faa940a0bac2, State: Initialized, Role: FOLLOWER
I20260806 02:53:22.397007 13064 raft_consensus.cc:729] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afca7f5211384dceab28249fd6b35176, State: Initialized, Role: FOLLOWER
I20260806 02:53:22.397049 13063 raft_consensus.cc:374] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:22.397095 13063 raft_consensus.cc:729] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ed219a3c264d8f860d6e82b3de0fca, State: Initialized, Role: FOLLOWER
I20260806 02:53:22.397583 13062 consensus_queue.cc:260] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.397583 13064 consensus_queue.cc:260] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.397915 13063 consensus_queue.cc:260] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.398823 12678 heartbeater.cc:503] Master 127.24.130.190:38563 was elected leader, sending a full tablet report...
I20260806 02:53:22.398841 13064 ts_tablet_manager.cc:1428] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 02:53:22.398844 13062 ts_tablet_manager.cc:1428] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 02:53:22.399171 13044 heartbeater.cc:503] Master 127.24.130.190:38563 was elected leader, sending a full tablet report...
I20260806 02:53:22.399187 12856 heartbeater.cc:503] Master 127.24.130.190:38563 was elected leader, sending a full tablet report...
I20260806 02:53:22.399564 13063 ts_tablet_manager.cc:1428] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260806 02:53:22.543195 12858 tablet.cc:2369] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:53:22.543964 12679 tablet.cc:2369] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:22.551200 13072 raft_consensus.cc:482] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:22.551594 13072 raft_consensus.cc:504] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.553643 13072 leader_election.cc:288] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813), afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797)
I20260806 02:53:22.563236 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.554299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47149 (local address 127.24.130.130:40813)
0806 02:53:22.554513 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.554518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.555000 (+ 482us) server_negotiation.cc:407] Connection header received
0806 02:53:22.555285 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.555289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.555396 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.555524 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:22.556225 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.557334 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.560669 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.561712 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.561978 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.562083 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.562841 (+ 758us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.562997 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:53:22.563236 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.554365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:38125 (local address 127.24.130.131:32797)
0806 02:53:22.554568 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.554583 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.554997 (+ 414us) server_negotiation.cc:407] Connection header received
0806 02:53:22.555305 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.555314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.555378 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.555523 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:22.556225 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.557334 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.560669 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.561712 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.561978 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.562082 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.562837 (+ 755us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.563001 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 02:53:22.563397 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.554074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.24.130.129:47149)
0806 02:53:22.554823 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.554851 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.555103 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.555600 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.555606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.555646 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:22.556014 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.556028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.557543 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.557547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.560372 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.560411 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.561977 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.561983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.562472 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.562577 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.562786 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.563038 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":670,"spinlock_wait_cycles":29952,"thread_start_us":233,"threads_started":1}
I20260806 02:53:22.563398 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.554118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32797 (local address 127.24.130.129:38125)
0806 02:53:22.554823 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.554851 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:22.555103 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:22.555600 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:22.555606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:22.555645 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:22.556014 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.556028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.557543 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.557547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.560364 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:22.560411 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.561977 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.561983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:22.562420 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.562564 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 02:53:22.562815 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:53:22.563038 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":710,"spinlock_wait_cycles":17024,"thread_start_us":215,"threads_started":1}
I20260806 02:53:22.564664 12976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f139416452c46d4b7cabd2d6b9d7818" candidate_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afca7f5211384dceab28249fd6b35176" is_pre_election: true
I20260806 02:53:22.564669 12799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f139416452c46d4b7cabd2d6b9d7818" candidate_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" is_pre_election: true
I20260806 02:53:22.565236 12799 raft_consensus.cc:2454] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ed219a3c264d8f860d6e82b3de0fca in term 0.
I20260806 02:53:22.565237 12976 raft_consensus.cc:2454] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ed219a3c264d8f860d6e82b3de0fca in term 0.
W20260806 02:53:22.566022 13045 tablet.cc:2369] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:22.566244 12518 leader_election.cc:302] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afca7f5211384dceab28249fd6b35176, b0ed219a3c264d8f860d6e82b3de0fca; no voters:
I20260806 02:53:22.566848 13072 raft_consensus.cc:2781] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:53:22.567015 13072 raft_consensus.cc:482] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:22.567108 13072 raft_consensus.cc:3037] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:22.568953 13072 raft_consensus.cc:504] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.569643 13072 leader_election.cc:288] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 1 election: Requested vote from peers 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813), afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797)
I20260806 02:53:22.570314 12799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f139416452c46d4b7cabd2d6b9d7818" candidate_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
I20260806 02:53:22.570319 12976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f139416452c46d4b7cabd2d6b9d7818" candidate_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afca7f5211384dceab28249fd6b35176"
I20260806 02:53:22.570561 12799 raft_consensus.cc:3037] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:22.570560 12976 raft_consensus.cc:3037] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:22.572474 12799 raft_consensus.cc:2454] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ed219a3c264d8f860d6e82b3de0fca in term 1.
I20260806 02:53:22.573123 12517 leader_election.cc:302] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20120fa7e3f54d3a9cf8faa940a0bac2, b0ed219a3c264d8f860d6e82b3de0fca; no voters:
I20260806 02:53:22.573366 12976 raft_consensus.cc:2454] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ed219a3c264d8f860d6e82b3de0fca in term 1.
I20260806 02:53:22.573449 13072 raft_consensus.cc:2781] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:22.573952 13072 raft_consensus.cc:686] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 LEADER]: Becoming Leader. State: Replica: b0ed219a3c264d8f860d6e82b3de0fca, State: Running, Role: LEADER
I20260806 02:53:22.574609 13072 consensus_queue.cc:237] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:22.579806 12411 catalog_manager.cc:5495] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129). New cstate: current_term: 1 leader_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } health_report { overall_health: UNKNOWN } } }
I20260806 02:53:22.619604 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.614419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.24.130.129:43247)
0806 02:53:22.614600 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.614605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.614634 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:53:22.614922 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.614926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.615016 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.615155 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:53:22.615478 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.616121 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.617296 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.617594 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.617724 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.618217 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:53:22.619093 (+ 876us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:53:22.619252 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.619337 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.619436 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:53:22.629781 12799 raft_consensus.cc:1261] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Refusing update from remote peer b0ed219a3c264d8f860d6e82b3de0fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:22.629786 12976 raft_consensus.cc:1261] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Refusing update from remote peer b0ed219a3c264d8f860d6e82b3de0fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:22.631194 13072 consensus_queue.cc:1035] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:22.631587 13075 consensus_queue.cc:1035] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:22.645051 13080 mvcc.cc:205] Tried to move back new op lower bound from 7315393751560159232 to 7315393751350276096. Current Snapshot: MvccSnapshot[applied={T|T < 7315393751560159232}]
I20260806 02:53:22.660934 12411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:22.661211 12411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:22.670300 12411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Sending DeleteTablet for 3 replicas of tablet 7f139416452c46d4b7cabd2d6b9d7818
I20260806 02:53:22.673909 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 12493
I20260806 02:53:22.673610 12773 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f139416452c46d4b7cabd2d6b9d7818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:22 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:55372
I20260806 02:53:22.679320 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f139416452c46d4b7cabd2d6b9d7818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:22 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:44230
I20260806 02:53:22.682134 13090 tablet_replica.cc:330] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: stopping tablet replica
I20260806 02:53:22.682979 13090 raft_consensus.cc:2229] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:22.683740 13090 raft_consensus.cc:2258] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:22.696328 13093 tablet_replica.cc:330] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: stopping tablet replica
I20260806 02:53:22.697021 13093 raft_consensus.cc:2229] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:22.697577 13090 ts_tablet_manager.cc:1905] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:22.697696 13093 raft_consensus.cc:2258] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:22.700423 13093 ts_tablet_manager.cc:1905] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:22.701407 13090 ts_tablet_manager.cc:1918] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:22.701658 13090 log.cc:1196] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/7f139416452c46d4b7cabd2d6b9d7818
I20260806 02:53:22.702265 13090 ts_tablet_manager.cc:1939] T 7f139416452c46d4b7cabd2d6b9d7818 P afca7f5211384dceab28249fd6b35176: Deleting consensus metadata
I20260806 02:53:22.705091 12394 catalog_manager.cc:4841] TS afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797): tablet 7f139416452c46d4b7cabd2d6b9d7818 (table test-table [id=031b523687c447f98d70bd68c9e408ea]) successfully deleted
I20260806 02:53:22.708308 13093 ts_tablet_manager.cc:1918] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:22.708709 13093 log.cc:1196] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/7f139416452c46d4b7cabd2d6b9d7818
I20260806 02:53:22.709630 13093 ts_tablet_manager.cc:1939] T 7f139416452c46d4b7cabd2d6b9d7818 P 20120fa7e3f54d3a9cf8faa940a0bac2: Deleting consensus metadata
W20260806 02:53:22.711772 12394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:43247 (error 108)
I20260806 02:53:22.712035 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 12684
I20260806 02:53:22.721141 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.720319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43247 (local address 127.0.0.1:33342)
0806 02:53:22.720678 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.720864 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 02:53:22.721814 12394 catalog_manager.cc:4584] TS b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f139416452c46d4b7cabd2d6b9d7818: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43247: connect: Connection refused (error 111)
I20260806 02:53:22.737692 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 12866
I20260806 02:53:22.750411 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.737909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.0.0.1:55382)
0806 02:53:22.750053 (+ 12144us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:22.750168 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.130:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11994}
I20260806 02:53:22.757989 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 12366
I20260806 02:53:22.780001 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38563
--webserver_interface=127.24.130.190
--webserver_port=36925
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:22.986944 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:22.987591 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:22.987651 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:22.994340 13096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:22.994509 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:22.994549 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:22.994573 13096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:22.994599 13096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:23.002247 13096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:23.004004 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:23.005347 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:23.031183 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.031178 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.031316 13107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.031754 13096 server_base.cc:1034] running on GCE node
I20260806 02:53:23.033119 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:23.034603 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:23.035848 13096 hybrid_clock.cc:648] HybridClock initialized: now 1785984803035813 us; error 55 us; skew 500 ppm
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:23.039587 13096 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:23.041380 13096 webserver.cc:460] Webserver started at http://127.24.130.190:36925/ using document root <none> and password file <none>
I20260806 02:53:23.042112 13096 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:23.042198 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:23.046940 13096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:23.049619 13121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.051796 13096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:23.052006 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "ca1b588a06cf432798fd24470d3de815"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "e13bb2374e4db6753d206f817c4798a1"
server_key_iv: "5f30b20c9487e5891327bb6aee9b854c"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.052795 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:23.099082 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:23.104453 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:23.104749 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:23.120391 13096 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:38563
I20260806 02:53:23.121001 13194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:38563 every 8 connection(s)
I20260806 02:53:23.123245 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:23.124271 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 13096
I20260806 02:53:23.125448 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:43247
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:38563
--webserver_port=45661
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:23.132427 13196 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:53:23.135700 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap starting.
I20260806 02:53:23.163533 13196 log.cc:824] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:23.180106 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:23.180862 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap complete.
I20260806 02:53:23.188724 13196 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:23.189437 13196 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1b588a06cf432798fd24470d3de815, State: Initialized, Role: FOLLOWER
I20260806 02:53:23.190423 13196 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:23.190608 13196 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:23.190819 13196 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:23.191012 13196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:53:23.195050 13196 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:23.195626 13196 leader_election.cc:302] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1b588a06cf432798fd24470d3de815; no voters:
I20260806 02:53:23.196314 13196 leader_election.cc:288] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:53:23.196537 13200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:53:23.196931 13200 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 LEADER]: Becoming Leader. State: Replica: ca1b588a06cf432798fd24470d3de815, State: Running, Role: LEADER
I20260806 02:53:23.197718 13200 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:23.199146 13196 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:23.200821 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca1b588a06cf432798fd24470d3de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } } }
I20260806 02:53:23.201100 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:23.202324 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1b588a06cf432798fd24470d3de815. Latest consensus state: current_term: 2 leader_uuid: "ca1b588a06cf432798fd24470d3de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } } }
I20260806 02:53:23.202966 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:23.203058 13210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:23.216722 13210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:23.218480 13210 catalog_manager.cc:1259] Loaded cluster ID: 50850d2b26a54e8d89881c7857e3d9c3
I20260806 02:53:23.218730 13210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:23.222381 13210 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 02:53:23.230867 13210 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 02:53:23.246626 13210 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:23.248034 13210 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Loaded TSK: 0
I20260806 02:53:23.249475 13210 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:53:23.320312 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:23.320580 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:23.320626 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:23.324841 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:23.324926 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:23.324990 13198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:53:23.330710 13198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:43247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=45661
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:23.331882 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:23.332979 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:23.343279 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.343263 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.343525 13198 server_base.cc:1034] running on GCE node
W20260806 02:53:23.343310 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.343986 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:23.344755 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:23.345949 13198 hybrid_clock.cc:648] HybridClock initialized: now 1785984803345923 us; error 52 us; skew 500 ppm
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:23.349156 13198 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:53:23.350353 13198 webserver.cc:460] Webserver started at http://127.24.130.129:45661/ using document root <none> and password file <none>
I20260806 02:53:23.351042 13198 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:23.351161 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:23.355378 13198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:53:23.358006 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.359416 13198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:53:23.359576 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "b0ed219a3c264d8f860d6e82b3de0fca"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "cdb3647652cad051536b65a094d65c16"
server_key_iv: "fb76aa30db5cecf44f7ad86fed93b11b"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.360201 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:23.386216 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:23.389556 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:23.389810 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:23.390478 13198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:23.391667 13246 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:53:23.396214 13198 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:53:23.396308 13198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 02:53:23.396420 13198 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:53:23.398751 13198 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:53:23.398867 13198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.003s sys 0.000s
I20260806 02:53:23.399151 13246 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap starting.
I20260806 02:53:23.411104 13198 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:43247
I20260806 02:53:23.411150 13387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:43247 every 8 connection(s)
I20260806 02:53:23.412601 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:23.420131 13246 log.cc:824] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:23.420756 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 13198
I20260806 02:53:23.421324 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:40813
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:38563
--webserver_port=38941
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:23.429013 13246 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:23.429492 13246 tablet_bootstrap.cc:492] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap complete.
I20260806 02:53:23.430823 13246 ts_tablet_manager.cc:1397] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260806 02:53:23.431692 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.415021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:40579 (local address 127.24.130.190:38563)
0806 02:53:23.415410 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.415432 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.415757 (+ 325us) server_negotiation.cc:407] Connection header received
0806 02:53:23.416932 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.416943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.417235 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.417571 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.418854 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.419833 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.421477 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.422218 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.422413 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.425361 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.425367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.425391 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:23.425427 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.427677 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.428334 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.428339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.428345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.428502 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.428982 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.428985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.428987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.429504 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.429685 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.430421 (+ 736us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.430775 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
I20260806 02:53:23.431783 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.415033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:40579)
0806 02:53:23.415578 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.415623 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.416497 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.417622 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.417676 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.418137 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:23.418596 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.418612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.420012 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.420015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.421319 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.421334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.422397 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.422404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.422680 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.423471 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:23.423489 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:23.425145 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:23.427830 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.427836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.427847 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.428185 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.428627 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.428633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.428638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.428841 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.429682 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:23.429692 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:23.430030 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.430268 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.430704 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":143,"threads_started":1}
I20260806 02:53:23.434007 13246 raft_consensus.cc:348] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:23.434455 13246 raft_consensus.cc:729] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ed219a3c264d8f860d6e82b3de0fca, State: Initialized, Role: FOLLOWER
I20260806 02:53:23.434759 13388 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38563
I20260806 02:53:23.434993 13388 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:23.435096 13246 consensus_queue.cc:260] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } } peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } }
I20260806 02:53:23.435562 13388 heartbeater.cc:511] Master 127.24.130.190:38563 requested a full tablet report, sending...
I20260806 02:53:23.436156 13246 ts_tablet_manager.cc:1428] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:53:23.438269 13142 ts_manager.cc:194] Registered new tserver with Master: b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247)
I20260806 02:53:23.442955 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:40579
I20260806 02:53:23.445742 13388 heartbeater.cc:503] Master 127.24.130.190:38563 was elected leader, sending a full tablet report...
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:53:23.454372 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.441956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43247 (local address 127.0.0.1:33354)
0806 02:53:23.442604 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.442676 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.442813 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.444478 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.444484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.444945 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:23.445079 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.445088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.446534 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.446537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.447607 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.447617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.448276 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.448281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.448402 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.449009 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:23.449027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:23.450603 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:23.452539 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.452549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.452555 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.452796 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.453115 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.453119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.453121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.453212 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.453673 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:23.453680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:23.453811 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.453956 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.454178 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":195,"threads_started":1}
I20260806 02:53:23.454372 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.441970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.24.130.129:43247)
0806 02:53:23.443967 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.443979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.444005 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:53:23.444078 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.444083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.444300 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.444442 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.445208 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.446404 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.447725 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.448126 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.448248 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.450797 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.450806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.450817 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:23.450839 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.452412 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.452931 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.452936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.452937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.453015 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.453313 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.453317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.453318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.453557 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.453702 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.454054 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.454203 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1890,"thread_start_us":121,"threads_started":1}
I20260806 02:53:23.455592 13302 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f139416452c46d4b7cabd2d6b9d7818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:22 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:33354
I20260806 02:53:23.456489 13401 tablet_replica.cc:330] stopping tablet replica
I20260806 02:53:23.456895 13401 raft_consensus.cc:2229] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:23.457238 13401 raft_consensus.cc:2258] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:23.458451 13401 ts_tablet_manager.cc:1905] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:23.463353 13401 ts_tablet_manager.cc:1918] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:23.463572 13401 log.cc:1196] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/7f139416452c46d4b7cabd2d6b9d7818
I20260806 02:53:23.463941 13401 ts_tablet_manager.cc:1939] T 7f139416452c46d4b7cabd2d6b9d7818 P b0ed219a3c264d8f860d6e82b3de0fca: Deleting consensus metadata
I20260806 02:53:23.466177 13125 catalog_manager.cc:4841] TS b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247): tablet 7f139416452c46d4b7cabd2d6b9d7818 (table test-table [id=031b523687c447f98d70bd68c9e408ea]) successfully deleted
W20260806 02:53:23.552007 13397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:23.552284 13397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:23.552330 13397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:23.556751 13397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:23.556891 13397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:23.556994 13397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:53:23.563140 13397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=38941
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:23.564322 13397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:23.565407 13397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:23.575868 13408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.575928 13409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.575902 13412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.576124 13397 server_base.cc:1034] running on GCE node
I20260806 02:53:23.576604 13397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:23.577265 13397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:23.578465 13397 hybrid_clock.cc:648] HybridClock initialized: now 1785984803578431 us; error 52 us; skew 500 ppm
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:23.581346 13397 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:53:23.582226 13397 webserver.cc:460] Webserver started at http://127.24.130.130:38941/ using document root <none> and password file <none>
I20260806 02:53:23.582858 13397 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:23.582963 13397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:23.586920 13397 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:23.589015 13423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.590411 13397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:53:23.590574 13397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "ee2099cd278565a948ef1f7b13b2f54c"
server_key_iv: "98ab7942fd859d7c357401cdbf830677"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.591295 13397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:23.607324 13397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:23.610395 13397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:23.610625 13397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:23.611341 13397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:23.612293 13397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:23.612344 13397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.612408 13397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:23.612442 13397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.622042 13397 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:40813
I20260806 02:53:23.622068 13573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:40813 every 8 connection(s)
I20260806 02:53:23.623116 13397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:23.631574 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 13397
I20260806 02:53:23.632071 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:32797
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:38563
--webserver_port=36439
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:23.637501 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.625168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:54109 (local address 127.24.130.190:38563)
0806 02:53:23.625324 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.625328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.626204 (+ 876us) server_negotiation.cc:407] Connection header received
0806 02:53:23.627707 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.627711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.627776 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.627880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.629075 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.629614 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.630810 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.631057 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.631172 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.633802 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.633810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.633812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:23.633838 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.635411 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.635947 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.635951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.635953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.636037 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.636380 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.636384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.636385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.636578 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.636685 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.637187 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.637330 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:53:23.638397 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.625572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:54109)
0806 02:53:23.626049 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.626095 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.627489 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.628062 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.628069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.628450 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:23.628925 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.628938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.629748 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.629751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.630635 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.630646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.631215 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.631220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.631397 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.632067 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:23.632083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:23.633623 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:23.635537 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.635545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.635561 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.635836 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.636138 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.636141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.636143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.636284 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.636685 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:23.636692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:23.636950 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.637144 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.637495 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260806 02:53:23.639688 13574 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38563
I20260806 02:53:23.640036 13574 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:23.640741 13574 heartbeater.cc:511] Master 127.24.130.190:38563 requested a full tablet report, sending...
I20260806 02:53:23.642014 13142 ts_manager.cc:194] Registered new tserver with Master: 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813)
I20260806 02:53:23.642995 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:54109
W20260806 02:53:23.756515 13578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:23.756846 13578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:23.756904 13578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:23.761163 13578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:23.761231 13578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:23.761291 13578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:53:23.766630 13578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=36439
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:23.767779 13578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:23.768793 13578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:23.778818 13592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.778913 13590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.778818 13589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.779024 13578 server_base.cc:1034] running on GCE node
I20260806 02:53:23.779500 13578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:23.780220 13578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:23.781410 13578 hybrid_clock.cc:648] HybridClock initialized: now 1785984803781395 us; error 28 us; skew 500 ppm
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:23.784384 13578 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:53:23.785264 13578 webserver.cc:460] Webserver started at http://127.24.130.131:36439/ using document root <none> and password file <none>
I20260806 02:53:23.785808 13578 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:23.785888 13578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:23.789812 13578 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:23.791932 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.793165 13578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:53:23.793304 13578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "afca7f5211384dceab28249fd6b35176"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-3kk6"
server_key: "aa729f232fabb80e630008ab0979a816"
server_key_iv: "760623b62e85ba781de58373d93a81c8"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.793948 13578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:23.809739 13578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:23.813378 13578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:23.813594 13578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:23.814231 13578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:23.815148 13578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:23.815219 13578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.815294 13578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:23.815335 13578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.824441 13578 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:32797
I20260806 02:53:23.824501 13760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:32797 every 8 connection(s)
I20260806 02:53:23.825528 13578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:23.830241 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 13578
Aug 06 02:53:23 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984803, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:23.840413 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.830989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.24.130.190:38563)
0806 02:53:23.831293 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.831296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.831384 (+ 88us) server_negotiation.cc:407] Connection header received
0806 02:53:23.831522 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.831524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.831564 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.831643 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.832582 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.833340 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.834359 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.834754 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.834876 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.836281 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.836286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.836292 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:23.836340 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.838252 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.838844 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.838848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.838854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.838966 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.839276 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.839279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.839281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.839529 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.839759 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.840000 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.840134 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
I20260806 02:53:23.841521 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.827482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38863 (local address 127.24.130.190:38563)
0806 02:53:23.827622 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.827626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.828474 (+ 848us) server_negotiation.cc:407] Connection header received
0806 02:53:23.829916 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.829920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.830006 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.830109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.831589 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.832453 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.833920 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.834256 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.834422 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.837383 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.837386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.837389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:23.837434 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.839078 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.839723 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.839727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.839729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.839819 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:23.840263 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:23.840267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:23.840269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:23.840456 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.840567 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.841209 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.841353 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":52}
I20260806 02:53:23.842710 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.827864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:38863)
0806 02:53:23.828320 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.828367 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.829683 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.830287 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.830295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.830760 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:23.831452 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.831474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.832595 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.832598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.833798 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.833808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.834415 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.834421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.834614 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.835388 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:23.835404 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:23.837203 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:23.839211 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.839216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.839242 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.839626 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.839922 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:23.839926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:23.839929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:23.840124 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:23.840570 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:23.840580 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:23.840902 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.841189 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.841604 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260806 02:53:23.843712 13761 heartbeater.cc:347] Connected to a master server at 127.24.130.190:38563
I20260806 02:53:23.843969 13761 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:23.844458 13761 heartbeater.cc:511] Master 127.24.130.190:38563 requested a full tablet report, sending...
I20260806 02:53:23.845393 13142 ts_manager.cc:194] Registered new tserver with Master: afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797)
I20260806 02:53:23.845945 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:38863
I20260806 02:53:23.854344 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:53:23.859083 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.855122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.24.130.190:38563)
0806 02:53:23.855268 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.855273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.855477 (+ 204us) server_negotiation.cc:407] Connection header received
0806 02:53:23.855626 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.855628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.855735 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.855826 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:53:23.856050 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.856612 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.857587 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.857835 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.857938 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.858216 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:53:23.858636 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:53:23.858809 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.858865 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.858915 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:53:23.859757 13142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:53:23.863155 13142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:53:23.871543 13302 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac3437715384874a7bcedc1bad27fad (DEFAULT_TABLE table=test-table [id=4ad917169f984d19a954bd6faa6ffcad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:23.872344 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac3437715384874a7bcedc1bad27fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:23.875535 13246 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap starting.
I20260806 02:53:23.876525 13246 tablet_bootstrap.cc:654] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:23.876646 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.870258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32797 (local address 127.0.0.1:44242)
0806 02:53:23.870377 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.870396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.870480 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.871753 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.871770 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.871794 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:23.871926 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.871934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.873498 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.873501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.874819 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.874830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.876137 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.876144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.876258 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.876314 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.876395 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.876487 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 02:53:23.876747 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.870711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.24.130.131:32797)
0806 02:53:23.871217 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.871230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.871255 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:53:23.871345 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.871351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.871579 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.871781 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:23.872067 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.873353 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.874988 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.875999 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.876135 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.876283 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.876451 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.876572 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":165,"threads_started":1}
I20260806 02:53:23.877121 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.870579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.0.0.1:55394)
0806 02:53:23.870951 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.870967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.871058 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.873042 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.873047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.873071 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:23.873163 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.873169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.874300 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.874306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.875300 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.875312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.876511 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.876514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.876688 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.876863 (+ 175us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.876934 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.877001 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":161,"threads_started":1}
I20260806 02:53:23.878206 13246 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: No bootstrap required, opened a new log
I20260806 02:53:23.878311 13246 ts_tablet_manager.cc:1397] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:53:23.878825 13246 raft_consensus.cc:348] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.878937 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.870776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.24.130.130:40813)
0806 02:53:23.872410 (+ 1634us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.872424 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.872452 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:53:23.872531 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.872536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.872881 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.873041 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:23.873287 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.874167 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.875462 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.876186 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.876306 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.876448 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.877427 (+ 979us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.877552 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1522,"thread_start_us":113,"threads_started":1}
I20260806 02:53:23.878965 13246 raft_consensus.cc:374] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:23.879021 13246 raft_consensus.cc:729] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ed219a3c264d8f860d6e82b3de0fca, State: Initialized, Role: FOLLOWER
I20260806 02:53:23.879153 13246 consensus_queue.cc:260] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.879441 13246 ts_tablet_manager.cc:1428] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:53:23.880157 13666 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac3437715384874a7bcedc1bad27fad (DEFAULT_TABLE table=test-table [id=4ad917169f984d19a954bd6faa6ffcad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:23.881351 13666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac3437715384874a7bcedc1bad27fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:23.881755 13487 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac3437715384874a7bcedc1bad27fad (DEFAULT_TABLE table=test-table [id=4ad917169f984d19a954bd6faa6ffcad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:23.882836 13487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac3437715384874a7bcedc1bad27fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:23.890219 13786 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Bootstrap starting.
I20260806 02:53:23.890770 13788 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Bootstrap starting.
I20260806 02:53:23.893528 13786 tablet_bootstrap.cc:654] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:23.893528 13788 tablet_bootstrap.cc:654] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:23.894556 13788 log.cc:824] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:23.894556 13786 log.cc:824] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:23.896641 13786 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: No bootstrap required, opened a new log
I20260806 02:53:23.896677 13788 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: No bootstrap required, opened a new log
I20260806 02:53:23.896867 13786 ts_tablet_manager.cc:1397] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260806 02:53:23.896899 13788 ts_tablet_manager.cc:1397] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260806 02:53:23.900228 13786 raft_consensus.cc:348] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.900338 13788 raft_consensus.cc:348] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.900553 13786 raft_consensus.cc:374] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:23.900581 13788 raft_consensus.cc:374] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:23.900619 13786 raft_consensus.cc:729] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afca7f5211384dceab28249fd6b35176, State: Initialized, Role: FOLLOWER
I20260806 02:53:23.900631 13788 raft_consensus.cc:729] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20120fa7e3f54d3a9cf8faa940a0bac2, State: Initialized, Role: FOLLOWER
I20260806 02:53:23.901090 13786 consensus_queue.cc:260] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.901096 13788 consensus_queue.cc:260] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.901837 13761 heartbeater.cc:503] Master 127.24.130.190:38563 was elected leader, sending a full tablet report...
I20260806 02:53:23.901836 13574 heartbeater.cc:503] Master 127.24.130.190:38563 was elected leader, sending a full tablet report...
I20260806 02:53:23.902222 13786 ts_tablet_manager.cc:1428] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260806 02:53:23.902221 13788 ts_tablet_manager.cc:1428] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260806 02:53:23.914196 13390 tablet.cc:2369] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:23.962458 13791 raft_consensus.cc:482] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:23.962802 13791 raft_consensus.cc:504] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.964190 13791 leader_election.cc:288] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813), b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247)
I20260806 02:53:23.969710 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.964567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:39889 (local address 127.24.130.129:43247)
0806 02:53:23.964993 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.964996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.965010 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:53:23.965086 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.965089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.965169 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.965293 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:23.965786 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.966581 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.968471 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.969163 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.969279 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.969457 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.969494 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.969566 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":136,"threads_started":1}
I20260806 02:53:23.969707 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.964676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:58569 (local address 127.24.130.130:40813)
0806 02:53:23.964813 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.964818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.965095 (+ 277us) server_negotiation.cc:407] Connection header received
0806 02:53:23.965181 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.965184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.965241 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.965320 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:23.965786 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.966583 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.968472 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.969079 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.969227 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.969292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.969488 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.969566 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:53:23.969743 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.964481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43247 (local address 127.24.130.131:39889)
0806 02:53:23.964753 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.964773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.964900 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.965364 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.965368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.965414 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:23.965657 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.965663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.966724 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.966727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.968326 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.968338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.969266 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.969279 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.969372 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.969391 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.969451 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.969577 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":265,"spinlock_wait_cycles":11648}
I20260806 02:53:23.969763 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.964485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.24.130.131:58569)
0806 02:53:23.964985 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.965004 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.965087 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.965364 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.965368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.965413 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:23.965655 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.965663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.966730 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.966733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.968326 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.968338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.969204 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.969207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.969329 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.969362 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.969450 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.969577 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":234,"spinlock_wait_cycles":30080,"thread_start_us":208,"threads_started":1}
I20260806 02:53:23.970544 13516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac3437715384874a7bcedc1bad27fad" candidate_uuid: "afca7f5211384dceab28249fd6b35176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" is_pre_election: true
I20260806 02:53:23.970716 13328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac3437715384874a7bcedc1bad27fad" candidate_uuid: "afca7f5211384dceab28249fd6b35176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" is_pre_election: true
I20260806 02:53:23.970970 13328 raft_consensus.cc:2454] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afca7f5211384dceab28249fd6b35176 in term 0.
I20260806 02:53:23.970965 13516 raft_consensus.cc:2454] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afca7f5211384dceab28249fd6b35176 in term 0.
I20260806 02:53:23.971513 13606 leader_election.cc:302] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20120fa7e3f54d3a9cf8faa940a0bac2, afca7f5211384dceab28249fd6b35176, b0ed219a3c264d8f860d6e82b3de0fca; no voters:
I20260806 02:53:23.971796 13791 raft_consensus.cc:2781] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:53:23.971885 13791 raft_consensus.cc:482] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:23.971931 13791 raft_consensus.cc:3037] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:23.973131 13791 raft_consensus.cc:504] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.973483 13791 leader_election.cc:288] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [CANDIDATE]: Term 1 election: Requested vote from peers 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813), b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247)
I20260806 02:53:23.973862 13328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac3437715384874a7bcedc1bad27fad" candidate_uuid: "afca7f5211384dceab28249fd6b35176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ed219a3c264d8f860d6e82b3de0fca"
I20260806 02:53:23.973872 13516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac3437715384874a7bcedc1bad27fad" candidate_uuid: "afca7f5211384dceab28249fd6b35176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
I20260806 02:53:23.974009 13328 raft_consensus.cc:3037] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:23.974018 13516 raft_consensus.cc:3037] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:23.975250 13516 raft_consensus.cc:2454] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afca7f5211384dceab28249fd6b35176 in term 1.
I20260806 02:53:23.975260 13328 raft_consensus.cc:2454] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afca7f5211384dceab28249fd6b35176 in term 1.
I20260806 02:53:23.975574 13605 leader_election.cc:302] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20120fa7e3f54d3a9cf8faa940a0bac2, afca7f5211384dceab28249fd6b35176; no voters:
I20260806 02:53:23.975760 13791 raft_consensus.cc:2781] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:23.976007 13791 raft_consensus.cc:686] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 1 LEADER]: Becoming Leader. State: Replica: afca7f5211384dceab28249fd6b35176, State: Running, Role: LEADER
I20260806 02:53:23.976375 13791 consensus_queue.cc:237] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:23.979676 13142 catalog_manager.cc:5495] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 reported cstate change: term changed from 0 to 1, leader changed from <none> to afca7f5211384dceab28249fd6b35176 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "afca7f5211384dceab28249fd6b35176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } health_report { overall_health: UNKNOWN } } }
I20260806 02:53:24.001442 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.997531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.24.130.131:32797)
0806 02:53:23.997663 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.997670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.997710 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:53:23.997937 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.997940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.997999 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.998111 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:53:23.998415 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.999040 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.999963 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.000172 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.000254 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:24.000581 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:53:24.001035 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:53:24.001133 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:24.001202 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 02:53:24.001257 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:53:24.007822 13328 raft_consensus.cc:1261] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Refusing update from remote peer afca7f5211384dceab28249fd6b35176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:24.007846 13516 raft_consensus.cc:1261] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Refusing update from remote peer afca7f5211384dceab28249fd6b35176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:24.008664 13791 consensus_queue.cc:1035] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:24.008926 13798 consensus_queue.cc:1035] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:24.017863 13804 mvcc.cc:205] Tried to move back new op lower bound from 7315393757208047616 to 7315393757090033664. Current Snapshot: MvccSnapshot[applied={T|T < 7315393757208047616}]
I20260806 02:53:24.019608 13803 mvcc.cc:205] Tried to move back new op lower bound from 7315393757208047616 to 7315393757090033664. Current Snapshot: MvccSnapshot[applied={T|T < 7315393757208047616}]
I20260806 02:53:24.024930 13141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:24.025147 13141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:24.027949 13141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Sending DeleteTablet for 3 replicas of tablet 2ac3437715384874a7bcedc1bad27fad
I20260806 02:53:24.028810 13302 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac3437715384874a7bcedc1bad27fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:24 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:33354
I20260806 02:53:24.028873 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 13198
I20260806 02:53:24.028882 13487 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac3437715384874a7bcedc1bad27fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:24 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:55394
I20260806 02:53:24.035787 13666 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac3437715384874a7bcedc1bad27fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:24 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:44242
I20260806 02:53:24.037482 13813 tablet_replica.cc:330] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: stopping tablet replica
I20260806 02:53:24.037950 13813 raft_consensus.cc:2229] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:53:24.038450 13813 raft_consensus.cc:2258] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:24.040441 13813 ts_tablet_manager.cc:1905] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:53:24.043507 13125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:43247 (error 108)
I20260806 02:53:24.043607 13813 ts_tablet_manager.cc:1918] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:24.043713 13813 log.cc:1196] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/2ac3437715384874a7bcedc1bad27fad
I20260806 02:53:24.044111 13813 ts_tablet_manager.cc:1939] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Deleting consensus metadata
I20260806 02:53:24.044708 13812 tablet_replica.cc:330] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: stopping tablet replica
I20260806 02:53:24.045080 13812 raft_consensus.cc:2229] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:24.045496 13812 raft_consensus.cc:2258] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:24.045471 13125 catalog_manager.cc:4841] TS afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797): tablet 2ac3437715384874a7bcedc1bad27fad (table test-table [id=4ad917169f984d19a954bd6faa6ffcad]) successfully deleted
W20260806 02:53:24.046073 13777 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:24.044499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43247 (local address 127.0.0.1:33362)
0806 02:53:24.044633 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.044654 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:24.044758 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:24.045956 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:53:24.046293 13125 catalog_manager.cc:4584] TS b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ac3437715384874a7bcedc1bad27fad: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:53:24.047457 13812 ts_tablet_manager.cc:1905] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:24.047535 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 13397
I20260806 02:53:24.058537 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 13578
I20260806 02:53:24.061443 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.061083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.0.0.1:55396)
0806 02:53:24.061233 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.061350 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.130:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:53:24.070994 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 13096
I20260806 02:53:24.083130 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38563
--webserver_interface=127.24.130.190
--webserver_port=36925
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:24.239959 13817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:24.240306 13817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:24.240381 13817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:24.244745 13817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:24.244828 13817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:24.244849 13817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:24.244870 13817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:24.244905 13817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:24.250824 13817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:24.252049 13817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:24.253216 13817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:24.264216 13822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.264297 13825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.264242 13823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:24.264721 13817 server_base.cc:1034] running on GCE node
I20260806 02:53:24.265618 13817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:24.266847 13817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:24.268039 13817 hybrid_clock.cc:648] HybridClock initialized: now 1785984804268008 us; error 48 us; skew 500 ppm
Aug 06 02:53:24 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984804, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:24.271286 13817 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:24.272398 13817 webserver.cc:460] Webserver started at http://127.24.130.190:36925/ using document root <none> and password file <none>
I20260806 02:53:24.273532 13817 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:24.273650 13817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:24.278446 13817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:24.281051 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:24.284443 13817 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 02:53:24.284922 13817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "ca1b588a06cf432798fd24470d3de815"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "e13bb2374e4db6753d206f817c4798a1"
server_key_iv: "5f30b20c9487e5891327bb6aee9b854c"
server_key_version: "encryptionkey@0"
I20260806 02:53:24.285629 13817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:24.336009 13817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:24.341468 13817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:24.341748 13817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:24.355618 13817 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:38563
I20260806 02:53:24.356693 13896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:38563 every 8 connection(s)
I20260806 02:53:24.358143 13817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:24.365284 13898 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:53:24.366492 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 13817
I20260806 02:53:24.367177 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:43247
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:38563
--webserver_port=45661
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:24.368706 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap starting.
I20260806 02:53:24.390295 13898 log.cc:824] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:24.409065 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:24.409536 13898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815: Bootstrap complete.
I20260806 02:53:24.413977 13898 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:24.414414 13898 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1b588a06cf432798fd24470d3de815, State: Initialized, Role: FOLLOWER
I20260806 02:53:24.414984 13898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:24.415135 13898 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:24.415226 13898 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:24.415342 13898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:53:24.417405 13898 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:24.417749 13898 leader_election.cc:302] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1b588a06cf432798fd24470d3de815; no voters:
I20260806 02:53:24.418126 13898 leader_election.cc:288] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:53:24.418226 13903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:53:24.418473 13903 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [term 3 LEADER]: Becoming Leader. State: Replica: ca1b588a06cf432798fd24470d3de815, State: Running, Role: LEADER
I20260806 02:53:24.418826 13903 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } }
I20260806 02:53:24.419535 13898 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:24.419953 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1b588a06cf432798fd24470d3de815. Latest consensus state: current_term: 3 leader_uuid: "ca1b588a06cf432798fd24470d3de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } } }
I20260806 02:53:24.420086 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:24.420351 13904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ca1b588a06cf432798fd24470d3de815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1b588a06cf432798fd24470d3de815" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 38563 } } }
I20260806 02:53:24.420455 13904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1b588a06cf432798fd24470d3de815 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:24.420567 13907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:24.426087 13907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:24.426904 13907 catalog_manager.cc:1259] Loaded cluster ID: 50850d2b26a54e8d89881c7857e3d9c3
I20260806 02:53:24.426972 13907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 02:53:24.428176 13907 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff4ac2439ad google::LogMessage::Fail() at ??:0
@ 0x7ff4ac24688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff4ac2434de google::LogMessage::Flush() at ??:0
@ 0x7ff4ac244a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff4b6b82fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff4b6b83d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff4b6b80579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff4b6bc4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff4ae4953d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff4ae495c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff4ae4973f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff4ae486512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff4afaca6db start_thread at ??:0
@ 0x7ff4aad4161f clone at ??:0
W20260806 02:53:24.542829 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:24.543360 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:24.543452 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:24.550539 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:24.550683 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:24.550781 13900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:53:24.559991 13900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:43247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=45661
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:24.561548 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:24.562985 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:24.576045 13959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.576879 13960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.577267 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:24.578557 13900 server_base.cc:1034] running on GCE node
I20260806 02:53:24.579067 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:24.579840 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:24.581075 13900 hybrid_clock.cc:648] HybridClock initialized: now 1785984804581043 us; error 45 us; skew 500 ppm
Aug 06 02:53:24 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984804, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:24.584440 13900 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:53:24.585472 13900 webserver.cc:460] Webserver started at http://127.24.130.129:45661/ using document root <none> and password file <none>
I20260806 02:53:24.586084 13900 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:24.586153 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:24.590487 13900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:24.593143 13971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:24.594803 13900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:24.594959 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "b0ed219a3c264d8f860d6e82b3de0fca"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "cdb3647652cad051536b65a094d65c16"
server_key_iv: "fb76aa30db5cecf44f7ad86fed93b11b"
server_key_version: "encryptionkey@0"
I20260806 02:53:24.595574 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:24.607491 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:24.611356 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:24.611590 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:24.612269 13900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:24.613520 13979 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:53:24.619642 13900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:53:24.619796 13900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 02:53:24.619921 13900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:53:24.621817 13900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:53:24.621901 13900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:53:24.622223 13979 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap starting.
I20260806 02:53:24.633721 13900 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:43247
I20260806 02:53:24.634001 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:43247 every 8 connection(s)
I20260806 02:53:24.634966 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:24.638695 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 13900
I20260806 02:53:24.639250 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:40813
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:38563
--webserver_port=38941
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:24.648005 13979 log.cc:824] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:24.655633 13979 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:24.656075 13979 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Bootstrap complete.
I20260806 02:53:24.657238 13979 ts_tablet_manager.cc:1397] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.007s
I20260806 02:53:24.659921 13979 raft_consensus.cc:348] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:24.660401 13979 raft_consensus.cc:729] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ed219a3c264d8f860d6e82b3de0fca, State: Initialized, Role: FOLLOWER
I20260806 02:53:24.660924 13979 consensus_queue.cc:260] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:24.661955 13979 ts_tablet_manager.cc:1428] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260806 02:53:24.739374 14126 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:24.637064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:56103)
0806 02:53:24.637602 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.637670 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:24.638762 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:24.738327 (+ 99565us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":160,"threads_started":1}
W20260806 02:53:24.740118 14123 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:38563 (0 consecutive failures): Network error: Failed to ping master at 127.24.130.190:38563: Client connection negotiation failed: client connection to 127.24.130.190:38563: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:53:24.741001 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.740680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:47589)
0806 02:53:24.740854 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.740942 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 02:53:24.741889 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.741579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:38061)
0806 02:53:24.741733 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.741807 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:53:24.742086 14123 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:53:24.782328 14127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:24.782727 14127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:24.782790 14127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:24.787055 14127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:24.787132 14127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:24.787194 14127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:53:24.792865 14127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=38941
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:24.794024 14127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:24.795388 14127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:24.806337 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:24.806447 14127 server_base.cc:1034] running on GCE node
W20260806 02:53:24.806370 14138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.806361 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:24.807140 14127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:24.807847 14127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:24.809015 14127 hybrid_clock.cc:648] HybridClock initialized: now 1785984804808995 us; error 55 us; skew 500 ppm
Aug 06 02:53:24 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984804, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:24.812232 14127 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:53:24.813282 14127 webserver.cc:460] Webserver started at http://127.24.130.130:38941/ using document root <none> and password file <none>
I20260806 02:53:24.813870 14127 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:24.813956 14127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:24.818166 14127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:53:24.820631 14148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:24.821955 14127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:53:24.822109 14127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
format_stamp: "Formatted at 2026-08-06 02:53:21 on dist-test-slave-3kk6"
server_key: "ee2099cd278565a948ef1f7b13b2f54c"
server_key_iv: "98ab7942fd859d7c357401cdbf830677"
server_key_version: "encryptionkey@0"
I20260806 02:53:24.822829 14127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:24.847352 14127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:24.850601 14127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:24.850884 14127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:24.851588 14127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:24.852780 14159 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:53:24.857165 14127 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:53:24.857282 14127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260806 02:53:24.857383 14127 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:53:24.859349 14127 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:53:24.859424 14127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 02:53:24.859678 14159 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Bootstrap starting.
I20260806 02:53:24.865777 14159 log.cc:824] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:24.872165 14127 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:40813
I20260806 02:53:24.872218 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:40813 every 8 connection(s)
I20260806 02:53:24.873119 14127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:24.874729 14159 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:24.875149 14159 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Bootstrap complete.
I20260806 02:53:24.875500 14159 ts_tablet_manager.cc:1397] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260806 02:53:24.876883 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.875276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:51309)
0806 02:53:24.875837 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.875993 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
W20260806 02:53:24.877393 14305 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:38563 (0 consecutive failures): Network error: Failed to ping master at 127.24.130.190:38563: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
I20260806 02:53:24.878120 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.877841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:46477)
0806 02:53:24.877990 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.878049 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:53:24.878893 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.878585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:54237)
0806 02:53:24.878758 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.878812 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:53:24.879098 14305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:53:24.879511 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 14127
I20260806 02:53:24.879212 14159 raft_consensus.cc:348] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:24.879835 14159 raft_consensus.cc:729] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20120fa7e3f54d3a9cf8faa940a0bac2, State: Initialized, Role: FOLLOWER
I20260806 02:53:24.880028 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:32797
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:38563
--webserver_port=36439
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:24.880371 14159 consensus_queue.cc:260] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:24.881466 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.881241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:36737)
0806 02:53:24.881352 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.881407 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 02:53:24.881513 14159 ts_tablet_manager.cc:1428] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:53:24.882171 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.881954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:44755)
0806 02:53:24.882064 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.882114 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:53:25.003080 14310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:25.003381 14310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:25.003455 14310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:25.007665 14310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:25.007746 14310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:25.007843 14310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:53:25.013409 14310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=36439
--tserver_master_addrs=127.24.130.190:38563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:25.014591 14310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:25.015758 14310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:25.025590 14321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:25.025695 14323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:25.025590 14320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:25.025923 14310 server_base.cc:1034] running on GCE node
I20260806 02:53:25.026432 14310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:25.027213 14310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:25.028416 14310 hybrid_clock.cc:648] HybridClock initialized: now 1785984805028398 us; error 39 us; skew 500 ppm
Aug 06 02:53:25 dist-test-slave-3kk6 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984805, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:25.031319 14310 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:53:25.032285 14310 webserver.cc:460] Webserver started at http://127.24.130.131:36439/ using document root <none> and password file <none>
I20260806 02:53:25.032871 14310 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:25.032967 14310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:25.037002 14310 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:25.039222 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.040680 14310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:25.040827 14310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "afca7f5211384dceab28249fd6b35176"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-3kk6"
server_key: "aa729f232fabb80e630008ab0979a816"
server_key_iv: "760623b62e85ba781de58373d93a81c8"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.041440 14310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:25.058255 14310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:25.061594 14310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:25.061839 14310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:25.062508 14310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:25.063500 14310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:25.063573 14310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.063648 14310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:25.063688 14310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.073625 14310 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:32797
I20260806 02:53:25.073652 14493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:32797 every 8 connection(s)
I20260806 02:53:25.074781 14310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:25.078778 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 14310
I20260806 02:53:25.079025 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.077087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:48799)
0806 02:53:25.077591 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:25.078105 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
W20260806 02:53:25.079509 14495 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:38563 (0 consecutive failures): Network error: Failed to ping master at 127.24.130.190:38563: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
I20260806 02:53:25.080158 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.079910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:46633)
0806 02:53:25.080046 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:25.080098 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:53:25.080737 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.080576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:48199)
0806 02:53:25.080646 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:25.080688 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260806 02:53:25.080938 14495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:53:25.743429 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.742915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:54223)
0806 02:53:25.743241 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:25.743316 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260806 02:53:25.883461 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.883015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:35493)
0806 02:53:25.883283 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:25.883349 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260806 02:53:26.081948 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.081499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:33633)
0806 02:53:26.081769 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.081829 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260806 02:53:26.244505 14507 raft_consensus.cc:482] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:26.244832 14507 raft_consensus.cc:504] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:26.246026 14507 leader_election.cc:288] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797), 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813)
Aug 06 02:53:26 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984804, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.130@KRBTEST.COM
Aug 06 02:53:26 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984804, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.131@KRBTEST.COM
I20260806 02:53:26.263646 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.246301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:32797 (local address 127.24.130.129:35809)
0806 02:53:26.246705 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.246723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:26.246839 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:26.248426 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:26.248429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:26.248607 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:26.249085 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:26.249101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.250430 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.250433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:26.251485 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:26.251495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.252223 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.252227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:26.252424 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:26.256781 (+ 4357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:26.256805 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:26.258877 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:26.261061 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:26.261064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:26.261065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:26.261420 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:26.261783 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:26.261786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:26.261788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:26.261951 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:26.262614 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:26.262622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:26.262886 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 02:53:26.263066 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:53:26.263436 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":237,"spinlock_wait_cycles":30336,"thread_start_us":88,"threads_started":1}
I20260806 02:53:26.263684 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.246451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47657 (local address 127.24.130.130:40813)
0806 02:53:26.247039 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.247050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.247083 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:53:26.247275 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.247280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.247612 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.247875 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.249284 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.250271 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.251612 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.252073 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.252242 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:26.258785 (+ 6543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:26.258796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:26.258819 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:26.258861 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:26.260856 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:26.261553 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:26.261558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:26.261562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:26.261684 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:26.262060 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:26.262062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:26.262063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:26.262492 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:26.262638 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:26.263203 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 02:53:26.263477 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":86,"threads_started":1}
I20260806 02:53:26.263684 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.246301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:40813 (local address 127.24.130.129:47657)
0806 02:53:26.246781 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.246797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:26.246914 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:26.247993 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:26.248000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:26.248584 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:26.249085 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:26.249102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.250437 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.250440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:26.251484 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:26.251495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.252203 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.252207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:26.252376 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:26.256780 (+ 4404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:26.256800 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:26.258509 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:26.260998 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:26.261008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:26.261025 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:26.261416 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:26.261811 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:26.261814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:26.261815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:26.261948 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:26.262614 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:26.262622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:26.262884 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 02:53:26.263066 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 02:53:26.263518 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":398,"spinlock_wait_cycles":13056,"thread_start_us":80,"threads_started":1}
I20260806 02:53:26.265261 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.246563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:35809 (local address 127.24.130.131:32797)
0806 02:53:26.246933 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.246963 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.247012 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:53:26.247729 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.247734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.248107 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.248436 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.249278 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.250263 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.251606 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:26.252082 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:26.252258 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:26.259099 (+ 6841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:26.259104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:26.259114 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:26.259145 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:26.260916 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:26.261555 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:26.261559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:26.261563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:26.261682 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:26.262059 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:26.262062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:26.262068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:26.262493 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:26.262639 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:26.263197 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:53:26.263502 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260806 02:53:26.265591 14239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac3437715384874a7bcedc1bad27fad" candidate_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" is_pre_election: true
I20260806 02:53:26.265972 14239 raft_consensus.cc:2454] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ed219a3c264d8f860d6e82b3de0fca in term 1.
I20260806 02:53:26.267220 13974 leader_election.cc:302] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20120fa7e3f54d3a9cf8faa940a0bac2, b0ed219a3c264d8f860d6e82b3de0fca; no voters:
I20260806 02:53:26.267562 14507 raft_consensus.cc:2781] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:53:26.267678 14507 raft_consensus.cc:482] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:26.267738 14507 raft_consensus.cc:3037] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 1 FOLLOWER]: Advancing to term 2
W20260806 02:53:26.267845 13975 leader_election.cc:341] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797): Not found: Tablet not found: 2ac3437715384874a7bcedc1bad27fad
I20260806 02:53:26.270073 14507 raft_consensus.cc:504] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:26.270540 14507 leader_election.cc:288] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 2 election: Requested vote from peers afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797), 20120fa7e3f54d3a9cf8faa940a0bac2 (127.24.130.130:40813)
I20260806 02:53:26.271099 14239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac3437715384874a7bcedc1bad27fad" candidate_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2"
W20260806 02:53:26.271263 13975 leader_election.cc:341] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797): Not found: Tablet not found: 2ac3437715384874a7bcedc1bad27fad
I20260806 02:53:26.271281 14239 raft_consensus.cc:3037] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:53:26.273492 14239 raft_consensus.cc:2454] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ed219a3c264d8f860d6e82b3de0fca in term 2.
I20260806 02:53:26.273881 13974 leader_election.cc:302] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20120fa7e3f54d3a9cf8faa940a0bac2, b0ed219a3c264d8f860d6e82b3de0fca; no voters: afca7f5211384dceab28249fd6b35176
I20260806 02:53:26.274057 14507 raft_consensus.cc:2781] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:53:26.274245 14507 raft_consensus.cc:686] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [term 2 LEADER]: Becoming Leader. State: Replica: b0ed219a3c264d8f860d6e82b3de0fca, State: Running, Role: LEADER
I20260806 02:53:26.274559 14507 consensus_queue.cc:237] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:26.275028 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.274752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:60847)
0806 02:53:26.274898 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.274961 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 02:53:26.805711 13975 consensus_peers.cc:487] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca -> Peer afca7f5211384dceab28249fd6b35176 (127.24.130.131:32797): Couldn't send request to peer afca7f5211384dceab28249fd6b35176. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ac3437715384874a7bcedc1bad27fad. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:53:26.865453 14239 raft_consensus.cc:1261] T 2ac3437715384874a7bcedc1bad27fad P 20120fa7e3f54d3a9cf8faa940a0bac2 [term 2 FOLLOWER]: Refusing update from remote peer b0ed219a3c264d8f860d6e82b3de0fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 02:53:26.866283 14516 consensus_queue.cc:1035] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 02:53:26.868645 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.868221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:50081)
0806 02:53:26.868449 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.868531 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260806 02:53:26.871313 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.870898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:36789)
0806 02:53:26.871138 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.871207 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260806 02:53:26.871451 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.871188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:39497)
0806 02:53:26.871311 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.871367 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:53:27.083316 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.082848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:50833)
0806 02:53:27.083124 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.083198 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260806 02:53:27.317589 14526 ts_tablet_manager.cc:927] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Initiating tablet copy from peer b0ed219a3c264d8f860d6e82b3de0fca (127.24.130.129:43247)
I20260806 02:53:27.318292 14526 tablet_copy_client.cc:316] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: tablet copy: Beginning tablet copy session from remote peer at address 127.24.130.129:43247
Aug 06 02:53:27 dist-test-slave-3kk6 krb5kdc[12352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984805, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.129@KRBTEST.COM
I20260806 02:53:27.328953 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.318963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43247 (local address 127.24.130.131:42835)
0806 02:53:27.319240 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.319270 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:27.319467 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:27.320056 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:27.320061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:27.320451 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:27.320572 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:27.320577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:27.321571 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:27.321577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:27.322287 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:27.322298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:27.322976 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:27.322978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:27.323060 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:27.323686 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:27.323699 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:27.325171 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:27.327160 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:27.327171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:27.327179 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:27.327468 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:27.327772 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:27.327775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:27.327776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:27.327877 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:27.328405 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:27.328411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:27.328526 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 02:53:27.328650 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:53:27.328791 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260806 02:53:27.329041 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.319246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:42835 (local address 127.24.130.129:43247)
0806 02:53:27.319609 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:27.319620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:27.319645 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:53:27.319698 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:27.319703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:27.319900 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:27.320027 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:27.320683 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:27.321427 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:27.322410 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:27.322860 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:27.322952 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:27.325303 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:27.325313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:27.325327 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:27.325349 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:27.327047 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:27.327579 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:27.327584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:27.327585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:27.327669 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:27.327963 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:27.327965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:27.327966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:27.328269 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:27.328440 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:27.328734 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 02:53:27.328884 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":140,"threads_started":1}
I20260806 02:53:27.330134 14085 tablet_copy_service.cc:140] P b0ed219a3c264d8f860d6e82b3de0fca: Received BeginTabletCopySession request for tablet 2ac3437715384874a7bcedc1bad27fad from peer afca7f5211384dceab28249fd6b35176 ({username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:42835)
I20260806 02:53:27.330335 14085 tablet_copy_service.cc:161] P b0ed219a3c264d8f860d6e82b3de0fca: Beginning new tablet copy session on tablet 2ac3437715384874a7bcedc1bad27fad from peer afca7f5211384dceab28249fd6b35176 at {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:42835: session id = afca7f5211384dceab28249fd6b35176-2ac3437715384874a7bcedc1bad27fad
I20260806 02:53:27.331590 14085 tablet_copy_source_session.cc:205] T 2ac3437715384874a7bcedc1bad27fad P b0ed219a3c264d8f860d6e82b3de0fca: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:53:27.335882 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac3437715384874a7bcedc1bad27fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:27.344906 14526 tablet_copy_client.cc:791] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: tablet copy: Starting download of 0 data blocks...
I20260806 02:53:27.345158 14526 tablet_copy_client.cc:655] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: tablet copy: Starting download of 1 WAL segments...
I20260806 02:53:27.347061 14526 tablet_copy_client.cc:531] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:53:27.348791 14526 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Bootstrap starting.
I20260806 02:53:27.354198 14526 log.cc:824] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:27.361090 14526 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:27.361402 14526 tablet_bootstrap.cc:492] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Bootstrap complete.
I20260806 02:53:27.361706 14526 ts_tablet_manager.cc:1397] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260806 02:53:27.364074 14526 raft_consensus.cc:348] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:27.364324 14526 raft_consensus.cc:729] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: afca7f5211384dceab28249fd6b35176, State: Initialized, Role: FOLLOWER
I20260806 02:53:27.364696 14526 consensus_queue.cc:260] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afca7f5211384dceab28249fd6b35176" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 32797 } } peers { permanent_uuid: "20120fa7e3f54d3a9cf8faa940a0bac2" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 40813 } } peers { permanent_uuid: "b0ed219a3c264d8f860d6e82b3de0fca" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43247 } }
I20260806 02:53:27.365653 14526 ts_tablet_manager.cc:1428] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260806 02:53:27.366051 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.365765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:51147)
0806 02:53:27.365906 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.365982 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:53:27.366496 14085 tablet_copy_service.cc:342] P b0ed219a3c264d8f860d6e82b3de0fca: Request end of tablet copy session afca7f5211384dceab28249fd6b35176-2ac3437715384874a7bcedc1bad27fad received from {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:42835
I20260806 02:53:27.366638 14085 tablet_copy_service.cc:433] P b0ed219a3c264d8f860d6e82b3de0fca: ending tablet copy session afca7f5211384dceab28249fd6b35176-2ac3437715384874a7bcedc1bad27fad on tablet 2ac3437715384874a7bcedc1bad27fad with peer afca7f5211384dceab28249fd6b35176
I20260806 02:53:27.366849 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.366550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:34937)
0806 02:53:27.366739 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.366795 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 02:53:27.698488 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.697978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:42511)
0806 02:53:27.698296 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.698367 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
I20260806 02:53:27.698582 14425 raft_consensus.cc:1203] T 2ac3437715384874a7bcedc1bad27fad P afca7f5211384dceab28249fd6b35176 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 02:53:27.699684 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.699324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:33997)
0806 02:53:27.699571 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.699623 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260806 02:53:27.872694 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.872266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:51441)
0806 02:53:27.872531 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:27.872602 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20260806 02:53:28.699762 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:28.699290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:33553)
0806 02:53:28.699563 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:28.699633 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260806 02:53:28.700871 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:28.700436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:49583)
0806 02:53:28.700726 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:28.700783 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260806 02:53:28.874083 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:28.873535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:55605)
0806 02:53:28.873896 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:28.873979 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":91,"threads_started":1}
I20260806 02:53:29.701267 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.700827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:44387)
0806 02:53:29.701097 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:29.701166 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260806 02:53:29.702042 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.701602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:36647)
0806 02:53:29.701874 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:29.701944 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260806 02:53:29.702056 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.701841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:38189)
0806 02:53:29.701952 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:29.702002 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:53:29.875447 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.875049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:60657)
0806 02:53:29.875300 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:29.875357 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260806 02:53:30.703296 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:30.702867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:56475)
0806 02:53:30.703134 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:30.703193 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
I20260806 02:53:30.703476 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:30.703043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:34623)
0806 02:53:30.703297 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:30.703362 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260806 02:53:30.876708 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:30.876255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:39085)
0806 02:53:30.876541 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:30.876611 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260806 02:53:31.704581 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.704146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:57579)
0806 02:53:31.704424 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.704479 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260806 02:53:31.704674 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.704146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:41655)
0806 02:53:31.704500 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.704557 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":101,"threads_started":1}
I20260806 02:53:31.877982 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.877495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:43673)
0806 02:53:31.877795 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.877867 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260806 02:53:32.705885 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:32.705413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:47917)
0806 02:53:32.705714 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:32.705785 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":127,"threads_started":1}
I20260806 02:53:32.705885 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:32.705415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:44845)
0806 02:53:32.705713 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:32.705788 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":120,"threads_started":1}
I20260806 02:53:32.879374 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:32.878923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:47831)
0806 02:53:32.879210 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:32.879275 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260806 02:53:33.707501 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:33.706970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:59701)
0806 02:53:33.707307 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:33.707389 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260806 02:53:33.707543 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:33.706970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:34649)
0806 02:53:33.707390 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:33.707451 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":109,"threads_started":1}
I20260806 02:53:33.880723 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:33.880186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:51821)
0806 02:53:33.880539 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:33.880613 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260806 02:53:34.708992 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:34.708465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:57519)
0806 02:53:34.708801 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:34.708890 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
I20260806 02:53:34.708992 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:34.708515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:33931)
0806 02:53:34.708827 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:34.708889 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260806 02:53:34.882337 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:34.881873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:40905)
0806 02:53:34.882172 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:34.882239 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
I20260806 02:53:35.710489 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:35.710070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:52693)
0806 02:53:35.710333 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:35.710402 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260806 02:53:35.710480 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:35.709996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:35161)
0806 02:53:35.710305 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:35.710379 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":120,"threads_started":1}
I20260806 02:53:35.884070 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:35.883556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:40039)
0806 02:53:35.883889 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:35.883963 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260806 02:53:36.712116 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:36.711549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:56813)
0806 02:53:36.711959 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:36.712015 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":126,"threads_started":1}
I20260806 02:53:36.712141 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:36.711549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:51131)
0806 02:53:36.711918 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:36.711989 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260806 02:53:36.885577 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:36.885041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:40845)
0806 02:53:36.885393 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:36.885471 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
I20260806 02:53:37.713546 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:37.713050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:52223)
0806 02:53:37.713389 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:37.713456 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260806 02:53:37.713539 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:37.713050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:40383)
0806 02:53:37.713368 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:37.713439 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260806 02:53:37.887068 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:37.886551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:36605)
0806 02:53:37.886892 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:37.886964 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260806 02:53:38.715066 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:38.714513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:52801)
0806 02:53:38.714891 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:38.714961 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
I20260806 02:53:38.715086 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:38.714512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:57435)
0806 02:53:38.714940 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:38.714998 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":155,"threads_started":1}
I20260806 02:53:38.888456 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:38.887952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:42695)
0806 02:53:38.888273 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:38.888353 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260806 02:53:39.716624 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:39.716065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.131:51311)
0806 02:53:39.716420 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:39.716502 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260806 02:53:39.716624 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:39.716071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.129:40065)
0806 02:53:39.716420 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:39.716501 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260806 02:53:39.889917 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:39.889484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:38563 (local address 127.24.130.130:59267)
0806 02:53:39.889746 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:39.889805 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260806 02:53:40.089534 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 13900
I20260806 02:53:40.105929 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 14127
I20260806 02:53:40.120334 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 14310
2026-08-06T02:53:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20237 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:40 dist-test-slave-3kk6 krb5kdc[14594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:40 dist-test-slave-3kk6 krb5kdc[14594](info): set up 2 sockets
Aug 06 02:53:40 dist-test-slave-3kk6 krb5kdc[14594](info): commencing operation
krb5kdc: starting...
W20260806 02:53:41.368645 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:53:41 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:53:41.518904 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41015
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:41.640880 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:41.641250 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:41.641335 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:41.645579 14612 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:53:41.645706 14612 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:53:41.645737 14612 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:53:41.645763 14612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:41.645812 14612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:41.645857 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:41.645895 14612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:41.645932 14612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:41.651886 14612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:41.653136 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:41.654246 14612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:41.663383 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:41.663388 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:41.663383 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:41.663779 14612 server_base.cc:1034] running on GCE node
I20260806 02:53:41.664397 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:41.665611 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:41.666792 14612 hybrid_clock.cc:648] HybridClock initialized: now 1785984821666774 us; error 35 us; skew 500 ppm
Aug 06 02:53:41 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:41.669906 14612 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:53:41.671002 14612 webserver.cc:460] Webserver started at http://127.24.130.190:43915/ using document root <none> and password file <none>
I20260806 02:53:41.671625 14612 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:41.671731 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:41.672000 14612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:41.674252 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "23bd1f44f78140eabd9a552c240c450f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "68432a3744bac485276a181fade97079"
server_key_iv: "92c1d4e07ec6f6746a8b38821f0ac804"
server_key_version: "encryptionkey@0"
I20260806 02:53:41.674963 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "23bd1f44f78140eabd9a552c240c450f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "68432a3744bac485276a181fade97079"
server_key_iv: "92c1d4e07ec6f6746a8b38821f0ac804"
server_key_version: "encryptionkey@0"
I20260806 02:53:41.679217 14612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:53:41.681970 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:41.683406 14612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:41.683593 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "23bd1f44f78140eabd9a552c240c450f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "68432a3744bac485276a181fade97079"
server_key_iv: "92c1d4e07ec6f6746a8b38821f0ac804"
server_key_version: "encryptionkey@0"
I20260806 02:53:41.683732 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:41.698002 14612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:41.701877 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:41.702123 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:41.709978 14612 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:41015
I20260806 02:53:41.710297 14708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:41015 every 8 connection(s)
I20260806 02:53:41.711093 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:41.714039 14709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:41.717687 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 14612
I20260806 02:53:41.717828 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:53:41.718137 25098 external_mini_cluster.cc:1428] Setting key 4269001d6e90eeaf0d40323587c35a53
I20260806 02:53:41.720489 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap starting.
I20260806 02:53:41.723197 14709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:41.724071 14709 log.cc:824] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:41.726377 14709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: No bootstrap required, opened a new log
Aug 06 02:53:41 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:41.730501 14709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:53:41.730836 14709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:41.730935 14709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bd1f44f78140eabd9a552c240c450f, State: Initialized, Role: FOLLOWER
I20260806 02:53:41.731458 14709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:53:41.731638 14709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:41.731747 14709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:41.731854 14709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:41.732965 14709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:53:41.733338 14709 leader_election.cc:302] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23bd1f44f78140eabd9a552c240c450f; no voters:
I20260806 02:53:41.733714 14709 leader_election.cc:288] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:41.733842 14714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:41.733948 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:41.719720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.24.130.190:41015)
0806 02:53:41.720176 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:41.720192 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:41.720230 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:53:41.720960 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:41.720980 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:41.721332 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:41.721618 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:41.722642 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:41.723850 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:41.724941 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:41.725434 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:41.725612 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:41.728470 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:41.728475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:41.728490 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:41.728518 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:41.730426 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:41.731070 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:41.731079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:41.731086 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:41.731229 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:41.731529 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:41.731535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:41.731536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:41.732118 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:41.732347 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:41.732608 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:53:41.732933 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":147,"threads_started":1}
I20260806 02:53:41.734071 14714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 LEADER]: Becoming Leader. State: Replica: 23bd1f44f78140eabd9a552c240c450f, State: Running, Role: LEADER
I20260806 02:53:41.734426 14714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:53:41.735152 14709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:41.737164 14715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23bd1f44f78140eabd9a552c240c450f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } } }
I20260806 02:53:41.737317 14715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:41.737440 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bd1f44f78140eabd9a552c240c450f. Latest consensus state: current_term: 1 leader_uuid: "23bd1f44f78140eabd9a552c240c450f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } } }
I20260806 02:53:41.737985 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: This master's current role is: LEADER
W20260806 02:53:41.740255 14734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:53:41.740386 14734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:53:41.740466 14735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:41.741488 14735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:41.747915 14735 catalog_manager.cc:1347] Generated new cluster ID: 69f2e9ff4da347cb9cab4e1b1f562050
I20260806 02:53:41.748018 14735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:41.757922 14735 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:41.759346 14735 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:41.781674 14735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Generated new TSK 0
I20260806 02:53:41.782720 14735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:53:41.821374 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:41015
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:41.942339 14745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:41.942747 14745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:41.942806 14745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:41.947321 14745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:41.947424 14745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:41.947490 14745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:53:41.953428 14745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:41.954867 14745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:41.955946 14745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:53:41.966591 14745 server_base.cc:1034] running on GCE node
W20260806 02:53:41.966506 14752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:41.966503 14751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:41.966503 14754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:41.967420 14745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:41.968158 14745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:41.969378 14745 hybrid_clock.cc:648] HybridClock initialized: now 1785984821969367 us; error 36 us; skew 500 ppm
Aug 06 02:53:41 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:41.972661 14745 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:53:41.973709 14745 webserver.cc:460] Webserver started at http://127.24.130.129:35801/ using document root <none> and password file <none>
I20260806 02:53:41.974328 14745 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:41.974431 14745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:41.974746 14745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:41.977005 14745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "8f8239525993478c993f8414bf61c25f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "6c8c3e4eceb099a90398fa7c5fa530fc"
server_key_iv: "4f9fb4807c2f0a718837c05e24a69e40"
server_key_version: "encryptionkey@0"
I20260806 02:53:41.977628 14745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "8f8239525993478c993f8414bf61c25f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "6c8c3e4eceb099a90398fa7c5fa530fc"
server_key_iv: "4f9fb4807c2f0a718837c05e24a69e40"
server_key_version: "encryptionkey@0"
I20260806 02:53:41.981837 14745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 02:53:41.984577 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:41.986029 14745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:53:41.986219 14745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "8f8239525993478c993f8414bf61c25f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "6c8c3e4eceb099a90398fa7c5fa530fc"
server_key_iv: "4f9fb4807c2f0a718837c05e24a69e40"
server_key_version: "encryptionkey@0"
I20260806 02:53:41.986366 14745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:42.007427 14745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:42.011242 14745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:42.011490 14745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:42.012199 14745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:42.013127 14745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:42.013206 14745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.013276 14745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:42.013320 14745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.023487 14745 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:37641
I20260806 02:53:42.023506 14918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:37641 every 8 connection(s)
I20260806 02:53:42.024572 14745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:42.030201 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 14745
I20260806 02:53:42.030328 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:42.030717 25098 external_mini_cluster.cc:1428] Setting key 46a61464e49ab38329b2d056758f1ad6
Aug 06 02:53:42 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984821, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:42.041616 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.026712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:56351 (local address 127.24.130.190:41015)
0806 02:53:42.026871 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.026876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.027991 (+ 1115us) server_negotiation.cc:407] Connection header received
0806 02:53:42.028935 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.028938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.028992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.029086 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:42.030508 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.031238 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.032924 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.033158 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.033245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.037027 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:42.037031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:42.037033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:42.037060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:42.038966 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.039742 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.039745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.039747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.039836 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.040327 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.040329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.040330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.040531 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:42.040611 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.041348 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.041480 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:53:42.043068 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.027107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:56351)
0806 02:53:42.027838 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.027886 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.028692 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.029303 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.029311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.029709 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:42.030260 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.030319 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.031471 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.031475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.032775 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.032789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.033339 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.033346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.033619 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.034430 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:42.034459 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:42.036791 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:42.039116 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:42.039126 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:42.039150 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:42.039615 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:42.039962 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:42.039968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:42.039972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:42.040216 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:42.040652 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:42.040659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:42.041056 (+ 397us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.041309 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.041735 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:53:42.044661 14919 heartbeater.cc:347] Connected to a master server at 127.24.130.190:41015
I20260806 02:53:42.044965 14919 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:42.045586 14919 heartbeater.cc:511] Master 127.24.130.190:41015 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:53:42.047716 14662 ts_manager.cc:194] Registered new tserver with Master: 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641)
I20260806 02:53:42.048955 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:56351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:53:42.066308 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:41015
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:42.189473 14927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:42.189806 14927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:42.189885 14927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:42.194222 14927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:42.194314 14927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:42.194416 14927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:53:42.200193 14927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:42.201499 14927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:42.202615 14927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:42.213140 14936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:42.213140 14938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:42.213239 14935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:42.213443 14927 server_base.cc:1034] running on GCE node
I20260806 02:53:42.214069 14927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:42.214818 14927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:42.216019 14927 hybrid_clock.cc:648] HybridClock initialized: now 1785984822215982 us; error 51 us; skew 500 ppm
Aug 06 02:53:42 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:42.219202 14927 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:53:42.220221 14927 webserver.cc:460] Webserver started at http://127.24.130.130:45253/ using document root <none> and password file <none>
I20260806 02:53:42.220830 14927 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:42.220939 14927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:42.221186 14927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:42.223483 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "bc4bbac4292f1deb78d96f983f61032e"
server_key_iv: "7d2b4c8b4ada5bf04e134bc212353b33"
server_key_version: "encryptionkey@0"
I20260806 02:53:42.224102 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "bc4bbac4292f1deb78d96f983f61032e"
server_key_iv: "7d2b4c8b4ada5bf04e134bc212353b33"
server_key_version: "encryptionkey@0"
I20260806 02:53:42.228147 14927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:53:42.230729 14951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.232046 14927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:53:42.232192 14927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "bc4bbac4292f1deb78d96f983f61032e"
server_key_iv: "7d2b4c8b4ada5bf04e134bc212353b33"
server_key_version: "encryptionkey@0"
I20260806 02:53:42.232321 14927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:42.245101 14927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:42.248522 14927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:42.248755 14927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:42.249425 14927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:42.250325 14927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:42.250399 14927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.250473 14927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:42.250525 14927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.260577 14927 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37033
I20260806 02:53:42.260617 15106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37033 every 8 connection(s)
I20260806 02:53:42.261580 14927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:42.265081 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 14927
I20260806 02:53:42.265239 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:42.265594 25098 external_mini_cluster.cc:1428] Setting key 966190ee030537c152f345b2154b2904
Aug 06 02:53:42 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:53:42.279177 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.263666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:54979 (local address 127.24.130.190:41015)
0806 02:53:42.263830 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.263834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.265102 (+ 1268us) server_negotiation.cc:407] Connection header received
0806 02:53:42.266425 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.266431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.266497 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.266595 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:42.268495 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.269042 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.270720 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.271161 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.271317 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.274742 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:42.274750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:42.274756 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:42.274798 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:42.276640 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.277240 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.277246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.277248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.277355 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.277736 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.277740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.277742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.278009 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:42.278137 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.278834 (+ 697us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.278968 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:53:42.280117 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.264043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:54979)
0806 02:53:42.264880 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.264950 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.266119 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.266879 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.266887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.267477 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:42.268287 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.268314 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.269198 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.269205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.270523 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.270535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.271360 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.271364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.271537 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.272496 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:42.272515 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:42.274468 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:42.276789 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:42.276795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:42.276810 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:42.277117 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:42.277475 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:42.277477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:42.277478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:42.277621 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:42.278169 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:42.278177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:42.278478 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.278743 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.279092 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
I20260806 02:53:42.281721 15108 heartbeater.cc:347] Connected to a master server at 127.24.130.190:41015
I20260806 02:53:42.282001 15108 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:42.282555 15108 heartbeater.cc:511] Master 127.24.130.190:41015 requested a full tablet report, sending...
I20260806 02:53:42.283861 14662 ts_manager.cc:194] Registered new tserver with Master: dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033)
I20260806 02:53:42.284730 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:54979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:53:42.300025 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:41015
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:42.420560 15115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:42.420881 15115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:42.420929 15115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:42.425088 15115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:42.425184 15115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:42.425247 15115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:53:42.430938 15115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:53:42.432101 15115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:42.433126 15115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:53:42.443547 15115 server_base.cc:1034] running on GCE node
W20260806 02:53:42.443732 15124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:42.443787 15121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:42.443742 15122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:42.444345 15115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:42.445013 15115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:42.446180 15115 hybrid_clock.cc:648] HybridClock initialized: now 1785984822446165 us; error 33 us; skew 500 ppm
Aug 06 02:53:42 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:42.449307 15115 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:53:42.450325 15115 webserver.cc:460] Webserver started at http://127.24.130.131:42307/ using document root <none> and password file <none>
I20260806 02:53:42.450989 15115 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:42.451063 15115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:42.451247 15115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:42.453408 15115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "008867d79c6d475e82497472ef7ad932"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "6fd99e72b869462cb6293fbd053176ea"
server_key_iv: "5e9c31e2f090194cdaa03d93c0dad15e"
server_key_version: "encryptionkey@0"
I20260806 02:53:42.453972 15115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "008867d79c6d475e82497472ef7ad932"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "6fd99e72b869462cb6293fbd053176ea"
server_key_iv: "5e9c31e2f090194cdaa03d93c0dad15e"
server_key_version: "encryptionkey@0"
I20260806 02:53:42.457937 15115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:53:42.460537 15133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.461812 15115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:53:42.461963 15115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "008867d79c6d475e82497472ef7ad932"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "6fd99e72b869462cb6293fbd053176ea"
server_key_iv: "5e9c31e2f090194cdaa03d93c0dad15e"
server_key_version: "encryptionkey@0"
I20260806 02:53:42.462095 15115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:42.481033 15115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:42.484331 15115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:42.484565 15115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:42.485224 15115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:42.486115 15115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:42.486191 15115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.486269 15115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:42.486313 15115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:42.496286 15115 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:45193
I20260806 02:53:42.496326 15280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:45193 every 8 connection(s)
I20260806 02:53:42.497282 15115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:42.498719 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 15115
I20260806 02:53:42.498869 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:42.499188 25098 external_mini_cluster.cc:1428] Setting key 45f3b45892436c069c0315972f1b5cc0
Aug 06 02:53:42 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984822, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:53:42.511683 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.499606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:42581 (local address 127.24.130.190:41015)
0806 02:53:42.499807 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.499813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.501103 (+ 1290us) server_negotiation.cc:407] Connection header received
0806 02:53:42.501987 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.501990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.502043 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.502120 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:42.503450 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.504027 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.505182 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.505424 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.505518 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.508128 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:42.508135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:42.508137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:42.508164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:42.509487 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.510045 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.510048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.510049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.510130 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.510507 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.510510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.510511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.510792 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:42.510878 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.511425 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.511540 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:53:42.512729 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.500075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:42581)
0806 02:53:42.500935 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.500992 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.501800 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.502294 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.502301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.502749 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:42.503285 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.503300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.504147 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.504149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.505051 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.505060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.505566 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.505569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.505710 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.506304 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:42.506321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:42.507944 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:42.509629 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:42.509634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:42.509647 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:42.509947 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:42.510251 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:42.510254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:42.510255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:42.510413 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:42.510916 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:42.510922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:42.511198 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.511375 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.511702 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":194,"threads_started":1}
I20260806 02:53:42.513904 15281 heartbeater.cc:347] Connected to a master server at 127.24.130.190:41015
I20260806 02:53:42.514168 15281 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:42.514771 15281 heartbeater.cc:511] Master 127.24.130.190:41015 requested a full tablet report, sending...
I20260806 02:53:42.515882 14662 ts_manager.cc:194] Registered new tserver with Master: 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193)
I20260806 02:53:42.516479 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:42581
I20260806 02:53:42.525959 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:53:42.537448 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.528392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.24.130.190:41015)
0806 02:53:42.528587 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.528593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.528772 (+ 179us) server_negotiation.cc:407] Connection header received
0806 02:53:42.528952 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.528956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.529017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.529121 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:42.529981 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.530589 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.531596 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.531877 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.532005 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.533620 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:42.533624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:42.533627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:42.533655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:42.535146 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.535681 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.535698 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.535702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.535851 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:42.536222 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:42.536226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:42.536228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:42.536579 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:42.536746 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.537114 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.537255 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:53:42.540876 14662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:53:42.544301 14662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:53:42.560879 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.553897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:45193 (local address 127.0.0.1:35884)
0806 02:53:42.554439 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.554501 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.554642 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.556048 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.556056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.556078 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:42.556246 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.556254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.558091 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.558095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.559232 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.559242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.560397 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.560400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.560506 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.560533 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.560624 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.560683 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":2,"thread_start_us":160,"threads_started":1}
I20260806 02:53:42.560928 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.555017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.24.130.130:37033)
0806 02:53:42.555535 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.555548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.555573 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:53:42.555652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.555657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.555882 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.556048 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:42.556632 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.557756 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.559375 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.559981 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.560128 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.560275 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.560665 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.560775 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":162,"threads_started":1}
I20260806 02:53:42.560720 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.554288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37033 (local address 127.0.0.1:48384)
0806 02:53:42.555112 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.555130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.555240 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.556243 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.556246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.556276 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:42.556499 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.556505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.557922 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.557925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.559235 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.559245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.560117 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.560120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.560228 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.560266 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.560386 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.560524 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":218,"spinlock_wait_cycles":9344,"thread_start_us":119,"threads_started":1}
I20260806 02:53:42.561710 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.554095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.24.130.131:45193)
0806 02:53:42.555535 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.555548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.555573 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:53:42.555652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.555657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.555884 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.556048 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:42.556414 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.557639 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.559394 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.560208 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.560378 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.560514 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.561440 (+ 926us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.561548 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":132,"threads_started":1}
I20260806 02:53:42.561877 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.555059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.24.130.129:37641)
0806 02:53:42.555512 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.555518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.555741 (+ 223us) server_negotiation.cc:407] Connection header received
0806 02:53:42.555879 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.555885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.556074 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.556219 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:42.557596 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.558781 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.560156 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.560996 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.561123 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.561251 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.561604 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.561707 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":137,"threads_started":1}
I20260806 02:53:42.562057 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.554775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:37641 (local address 127.0.0.1:60198)
0806 02:53:42.555625 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.555636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.555758 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.556243 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.556246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.556268 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:42.556488 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.556498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.558909 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.558913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.559999 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.560010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.561192 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.561196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.561299 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.561327 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.561425 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.561503 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":34,"spinlock_wait_cycles":1280,"thread_start_us":114,"threads_started":1}
I20260806 02:53:42.564651 15012 tablet_service.cc:1467] Processing CreateTablet for tablet 10d1c6a45caf4470a8b10409dd52e07b (DEFAULT_TABLE table=test-table [id=0c8a189e2daa4d638ccfed7804bd0409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:42.564764 15195 tablet_service.cc:1467] Processing CreateTablet for tablet 10d1c6a45caf4470a8b10409dd52e07b (DEFAULT_TABLE table=test-table [id=0c8a189e2daa4d638ccfed7804bd0409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:42.564764 14836 tablet_service.cc:1467] Processing CreateTablet for tablet 10d1c6a45caf4470a8b10409dd52e07b (DEFAULT_TABLE table=test-table [id=0c8a189e2daa4d638ccfed7804bd0409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:42.565867 15012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d1c6a45caf4470a8b10409dd52e07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:42.565871 14836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d1c6a45caf4470a8b10409dd52e07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:42.565884 15195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d1c6a45caf4470a8b10409dd52e07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:42.572706 15303 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Bootstrap starting.
I20260806 02:53:42.574018 15304 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Bootstrap starting.
I20260806 02:53:42.574220 15307 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Bootstrap starting.
I20260806 02:53:42.576095 15304 tablet_bootstrap.cc:654] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:42.576156 15303 tablet_bootstrap.cc:654] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:42.576397 15307 tablet_bootstrap.cc:654] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:42.577173 15303 log.cc:824] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:42.577190 15307 log.cc:824] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:42.577239 15304 log.cc:824] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:42.579334 15307 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: No bootstrap required, opened a new log
I20260806 02:53:42.579456 15303 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: No bootstrap required, opened a new log
I20260806 02:53:42.579598 15307 ts_tablet_manager.cc:1397] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:53:42.579654 15303 ts_tablet_manager.cc:1397] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260806 02:53:42.579675 15304 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: No bootstrap required, opened a new log
I20260806 02:53:42.579831 15304 ts_tablet_manager.cc:1397] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:53:42.583513 15307 raft_consensus.cc:348] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.583837 15307 raft_consensus.cc:374] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:42.583704 15304 raft_consensus.cc:348] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.583704 15303 raft_consensus.cc:348] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.583928 15307 raft_consensus.cc:729] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8239525993478c993f8414bf61c25f, State: Initialized, Role: FOLLOWER
I20260806 02:53:42.583987 15304 raft_consensus.cc:374] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:42.583987 15303 raft_consensus.cc:374] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:42.584050 15303 raft_consensus.cc:729] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4af35ab0014bc3b0fbc67947c2bb03, State: Initialized, Role: FOLLOWER
I20260806 02:53:42.584054 15304 raft_consensus.cc:729] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008867d79c6d475e82497472ef7ad932, State: Initialized, Role: FOLLOWER
I20260806 02:53:42.584456 15307 consensus_queue.cc:260] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.584494 15303 consensus_queue.cc:260] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.584494 15304 consensus_queue.cc:260] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.585584 15303 ts_tablet_manager.cc:1428] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 02:53:42.585584 15304 ts_tablet_manager.cc:1428] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:53:42.585584 15307 ts_tablet_manager.cc:1428] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:53:42.585695 15281 heartbeater.cc:503] Master 127.24.130.190:41015 was elected leader, sending a full tablet report...
I20260806 02:53:42.585848 14919 heartbeater.cc:503] Master 127.24.130.190:41015 was elected leader, sending a full tablet report...
I20260806 02:53:42.586927 15108 heartbeater.cc:503] Master 127.24.130.190:41015 was elected leader, sending a full tablet report...
I20260806 02:53:42.733582 15315 raft_consensus.cc:482] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:42.733887 15315 raft_consensus.cc:504] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.735352 15315 leader_election.cc:288] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193), dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033)
I20260806 02:53:42.740702 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.735797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:60867 (local address 127.24.130.131:45193)
0806 02:53:42.736000 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.736006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.736413 (+ 407us) server_negotiation.cc:407] Connection header received
0806 02:53:42.736738 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.736741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.736790 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.736895 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:42.737298 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.737902 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.739494 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.739950 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.740100 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.740175 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.740455 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.740542 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:53:42.740712 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.735797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:41453 (local address 127.24.130.130:37033)
0806 02:53:42.736004 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.736008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.736409 (+ 401us) server_negotiation.cc:407] Connection header received
0806 02:53:42.736641 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.736644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.736769 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.736895 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:42.737294 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.737966 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.739493 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.739945 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.740100 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.740175 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.740451 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.740543 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 02:53:42.740757 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.735725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37033 (local address 127.24.130.129:41453)
0806 02:53:42.736258 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.736278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.736492 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.736930 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.736934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.736956 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:42.737155 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.737164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.738098 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.738103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.739338 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.739349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.740100 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.740104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.740283 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.740327 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.740429 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.740551 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":86,"spinlock_wait_cycles":23168,"thread_start_us":134,"threads_started":1}
I20260806 02:53:42.740761 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.735800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:45193 (local address 127.24.130.129:60867)
0806 02:53:42.736282 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.736297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:42.736480 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:42.736912 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:42.736984 (+ 72us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:42.737013 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:42.737156 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.737162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.738034 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.738038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.739326 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:42.739346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.740100 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.740104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:42.740281 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.740320 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:53:42.740432 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:53:42.740542 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":88,"spinlock_wait_cycles":16256,"thread_start_us":138,"threads_started":1}
I20260806 02:53:42.741525 15041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1c6a45caf4470a8b10409dd52e07b" candidate_uuid: "8f8239525993478c993f8414bf61c25f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" is_pre_election: true
I20260806 02:53:42.741714 15223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1c6a45caf4470a8b10409dd52e07b" candidate_uuid: "8f8239525993478c993f8414bf61c25f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "008867d79c6d475e82497472ef7ad932" is_pre_election: true
I20260806 02:53:42.741978 15041 raft_consensus.cc:2454] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8239525993478c993f8414bf61c25f in term 0.
I20260806 02:53:42.742090 15223 raft_consensus.cc:2454] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8239525993478c993f8414bf61c25f in term 0.
I20260806 02:53:42.742558 14771 leader_election.cc:302] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f8239525993478c993f8414bf61c25f, dc4af35ab0014bc3b0fbc67947c2bb03; no voters:
I20260806 02:53:42.743108 15315 raft_consensus.cc:2781] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:53:42.743219 15315 raft_consensus.cc:482] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:42.743307 15315 raft_consensus.cc:3037] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:42.744554 15315 raft_consensus.cc:504] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.745004 15315 leader_election.cc:288] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 1 election: Requested vote from peers 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193), dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033)
I20260806 02:53:42.745553 15223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1c6a45caf4470a8b10409dd52e07b" candidate_uuid: "8f8239525993478c993f8414bf61c25f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "008867d79c6d475e82497472ef7ad932"
I20260806 02:53:42.745553 15041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d1c6a45caf4470a8b10409dd52e07b" candidate_uuid: "8f8239525993478c993f8414bf61c25f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
I20260806 02:53:42.745712 15223 raft_consensus.cc:3037] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:42.745716 15041 raft_consensus.cc:3037] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:42.747200 15223 raft_consensus.cc:2454] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8239525993478c993f8414bf61c25f in term 1.
I20260806 02:53:42.747195 15041 raft_consensus.cc:2454] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8239525993478c993f8414bf61c25f in term 1.
I20260806 02:53:42.747581 14771 leader_election.cc:302] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 008867d79c6d475e82497472ef7ad932, 8f8239525993478c993f8414bf61c25f, dc4af35ab0014bc3b0fbc67947c2bb03; no voters:
I20260806 02:53:42.747807 15315 raft_consensus.cc:2781] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:42.748123 15315 raft_consensus.cc:686] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 LEADER]: Becoming Leader. State: Replica: 8f8239525993478c993f8414bf61c25f, State: Running, Role: LEADER
W20260806 02:53:42.748373 15282 tablet.cc:2369] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:42.748597 15315 consensus_queue.cc:237] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:53:42.752336 14662 catalog_manager.cc:5495] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f8239525993478c993f8414bf61c25f (127.24.130.129). New cstate: current_term: 1 leader_uuid: "8f8239525993478c993f8414bf61c25f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } health_report { overall_health: UNKNOWN } } }
W20260806 02:53:42.762980 15109 tablet.cc:2369] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:53:42.776175 14921 tablet.cc:2369] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:42.798388 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.794512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.24.130.129:37641)
0806 02:53:42.794705 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:42.794710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:42.794729 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:53:42.794936 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:42.794939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:42.795000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:42.795096 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:53:42.795351 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.795867 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.796797 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:42.797031 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:42.797118 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:42.797485 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:53:42.797966 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:53:42.798063 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:42.798197 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 02:53:42.798259 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:53:42.804852 15223 raft_consensus.cc:1261] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Refusing update from remote peer 8f8239525993478c993f8414bf61c25f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:42.804885 15041 raft_consensus.cc:1261] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Refusing update from remote peer 8f8239525993478c993f8414bf61c25f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:42.805662 15315 consensus_queue.cc:1035] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [LEADER]: Connected to new peer: Peer: permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:42.805974 15319 consensus_queue.cc:1035] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:42.813752 15324 mvcc.cc:205] Tried to move back new op lower bound from 7315393834200780800 to 7315393833981435904. Current Snapshot: MvccSnapshot[applied={T|T < 7315393834200780800}]
I20260806 02:53:42.824966 14662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:42.825167 14662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:42.828125 14662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Sending DeleteTablet for 3 replicas of tablet 10d1c6a45caf4470a8b10409dd52e07b
I20260806 02:53:42.829128 15012 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d1c6a45caf4470a8b10409dd52e07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:42 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48384
I20260806 02:53:42.829221 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 14745
I20260806 02:53:42.829792 15195 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d1c6a45caf4470a8b10409dd52e07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:42 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:35884
I20260806 02:53:42.831849 15341 tablet_replica.cc:330] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: stopping tablet replica
I20260806 02:53:42.832286 15341 raft_consensus.cc:2229] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:42.832720 15341 raft_consensus.cc:2258] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:42.835695 15341 ts_tablet_manager.cc:1905] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:42.837535 15340 tablet_replica.cc:330] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: stopping tablet replica
I20260806 02:53:42.837965 15340 raft_consensus.cc:2229] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:42.838395 15340 raft_consensus.cc:2258] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:42.838806 15341 ts_tablet_manager.cc:1918] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:42.838918 15341 log.cc:1196] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/10d1c6a45caf4470a8b10409dd52e07b
I20260806 02:53:42.839282 15341 ts_tablet_manager.cc:1939] T 10d1c6a45caf4470a8b10409dd52e07b P 008867d79c6d475e82497472ef7ad932: Deleting consensus metadata
I20260806 02:53:42.840806 14638 catalog_manager.cc:4841] TS 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193): tablet 10d1c6a45caf4470a8b10409dd52e07b (table test-table [id=0c8a189e2daa4d638ccfed7804bd0409]) successfully deleted
I20260806 02:53:42.844246 15340 ts_tablet_manager.cc:1905] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:53:42.846915 14638 connection.cc:535] client connection to 127.24.130.129:37641 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 02:53:42.847103 14638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:53:42.847325 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 14927
I20260806 02:53:42.851899 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.851558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:37641 (local address 127.0.0.1:60206)
0806 02:53:42.851727 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.851811 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:37641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:53:42.852159 14638 catalog_manager.cc:4584] TS 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10d1c6a45caf4470a8b10409dd52e07b: Network error: Client connection negotiation failed: client connection to 127.24.130.129:37641: connect: Connection refused (error 111)
I20260806 02:53:42.865240 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 15115
I20260806 02:53:42.871320 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:42.865432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37033 (local address 127.0.0.1:48400)
0806 02:53:42.871040 (+ 5608us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:42.871162 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.130:37033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5439}
I20260806 02:53:42.887866 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 14612
I20260806 02:54:02.900172 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41015
--webserver_interface=127.24.130.190
--webserver_port=43915
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:03.017472 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:03.017838 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:03.017946 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:03.022148 15343 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:54:03.022256 15343 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:54:03.022285 15343 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:54:03.022322 15343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:03.022367 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:03.022410 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:03.022449 15343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:03.022485 15343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:03.028264 15343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=43915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:03.029477 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:03.030656 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:03.045670 15350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.045679 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.045696 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:03.045740 15343 server_base.cc:1034] running on GCE node
I20260806 02:54:03.046697 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:03.047979 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:03.049175 15343 hybrid_clock.cc:648] HybridClock initialized: now 1785984843049154 us; error 38 us; skew 500 ppm
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:03.052348 15343 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:54:03.053398 15343 webserver.cc:460] Webserver started at http://127.24.130.190:43915/ using document root <none> and password file <none>
I20260806 02:54:03.053962 15343 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:03.054042 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:03.058173 15343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:54:03.060470 15364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.061774 15343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:54:03.061892 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "23bd1f44f78140eabd9a552c240c450f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "68432a3744bac485276a181fade97079"
server_key_iv: "92c1d4e07ec6f6746a8b38821f0ac804"
server_key_version: "encryptionkey@0"
I20260806 02:54:03.062534 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:03.090266 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:03.094098 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:03.094292 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:03.102367 15343 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:41015
I20260806 02:54:03.102699 15439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:41015 every 8 connection(s)
I20260806 02:54:03.103719 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:03.108383 15440 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:54:03.109174 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 15343
I20260806 02:54:03.109719 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:37641
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:41015
--webserver_port=35801
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:54:03.110839 15440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap starting.
I20260806 02:54:03.129300 15440 log.cc:824] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:03.138621 15440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:54:03.139117 15440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap complete.
I20260806 02:54:03.143399 15440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:03.143841 15440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bd1f44f78140eabd9a552c240c450f, State: Initialized, Role: FOLLOWER
I20260806 02:54:03.144377 15440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:03.144495 15440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:03.144601 15440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:03.144729 15440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:54:03.147007 15440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:03.147382 15440 leader_election.cc:302] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23bd1f44f78140eabd9a552c240c450f; no voters:
I20260806 02:54:03.147802 15440 leader_election.cc:288] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:54:03.147902 15444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:54:03.148095 15444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 LEADER]: Becoming Leader. State: Replica: 23bd1f44f78140eabd9a552c240c450f, State: Running, Role: LEADER
I20260806 02:54:03.148492 15444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:03.149183 15440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:03.150193 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bd1f44f78140eabd9a552c240c450f. Latest consensus state: current_term: 2 leader_uuid: "23bd1f44f78140eabd9a552c240c450f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } } }
I20260806 02:54:03.150327 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:03.150259 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23bd1f44f78140eabd9a552c240c450f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } } }
I20260806 02:54:03.150388 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:03.150750 15453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:03.156132 15453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:03.156924 15453 catalog_manager.cc:1259] Loaded cluster ID: 69f2e9ff4da347cb9cab4e1b1f562050
I20260806 02:54:03.157040 15453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:03.159708 15453 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:03.160655 15453 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Loaded TSK: 0
I20260806 02:54:03.175683 15453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Generated new TSK 1
I20260806 02:54:03.176011 15453 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:54:03.234432 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:03.234836 15442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:03.234918 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:03.239089 15442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:03.239158 15442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:03.239256 15442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:03.244693 15442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:37641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=35801
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:03.245756 15442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:03.246872 15442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:03.257179 15475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.257179 15474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.257220 15479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:03.257768 15442 server_base.cc:1034] running on GCE node
I20260806 02:54:03.258312 15442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:03.259069 15442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:03.260304 15442 hybrid_clock.cc:648] HybridClock initialized: now 1785984843260279 us; error 47 us; skew 500 ppm
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:03.263294 15442 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:54:03.264412 15442 webserver.cc:460] Webserver started at http://127.24.130.129:35801/ using document root <none> and password file <none>
I20260806 02:54:03.265012 15442 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:03.265096 15442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:03.269451 15442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:54:03.271853 15490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.273250 15442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:03.273394 15442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "8f8239525993478c993f8414bf61c25f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "6c8c3e4eceb099a90398fa7c5fa530fc"
server_key_iv: "4f9fb4807c2f0a718837c05e24a69e40"
server_key_version: "encryptionkey@0"
I20260806 02:54:03.273998 15442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:03.293723 15442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:03.297372 15442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:03.297608 15442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:03.298324 15442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:03.299562 15503 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:54:03.303925 15442 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:54:03.304000 15442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 02:54:03.304052 15442 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:54:03.305758 15442 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:54:03.305825 15442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:54:03.306093 15503 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Bootstrap starting.
I20260806 02:54:03.317641 15442 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:37641
I20260806 02:54:03.317749 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:37641 every 8 connection(s)
I20260806 02:54:03.319046 15442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:03.327419 15503 log.cc:824] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:03.328610 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 15442
I20260806 02:54:03.329135 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:37033
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:41015
--webserver_port=45253
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:03.336522 15503 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:54:03.337164 15503 tablet_bootstrap.cc:492] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Bootstrap complete.
I20260806 02:54:03.337175 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.321463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:57883 (local address 127.24.130.190:41015)
0806 02:54:03.321889 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.321932 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.322361 (+ 429us) server_negotiation.cc:407] Connection header received
0806 02:54:03.323601 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.323612 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.323929 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.324266 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:03.325473 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.326453 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.327867 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.328391 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.328585 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.331865 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:03.331870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:03.331884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:03.331916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:03.333736 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.334371 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.334376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.334380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.334500 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.334930 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.334934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.334935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.335289 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:03.335442 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.335997 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.336264 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":161,"threads_started":1}
I20260806 02:54:03.337190 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.321577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:57883)
0806 02:54:03.322168 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:03.322214 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:03.323149 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:03.324322 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:03.324340 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:03.324812 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:03.325274 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.325289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.326653 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.326658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.327678 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.327688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.328561 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.328565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.328761 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.329562 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:03.329583 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:03.331601 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:03.333870 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:03.333876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:03.333888 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:03.334246 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:03.334610 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:03.334614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:03.334616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:03.334809 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:03.335414 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:03.335420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:03.335672 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 02:54:03.335856 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 02:54:03.336220 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":173,"threads_started":1}
I20260806 02:54:03.338904 15503 ts_tablet_manager.cc:1397] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.015s
I20260806 02:54:03.339795 15651 heartbeater.cc:347] Connected to a master server at 127.24.130.190:41015
I20260806 02:54:03.340058 15651 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:03.340570 15651 heartbeater.cc:511] Master 127.24.130.190:41015 requested a full tablet report, sending...
I20260806 02:54:03.343020 15392 ts_manager.cc:194] Registered new tserver with Master: 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641)
I20260806 02:54:03.342886 15503 raft_consensus.cc:348] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:54:03.343554 15503 raft_consensus.cc:729] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8239525993478c993f8414bf61c25f, State: Initialized, Role: FOLLOWER
I20260806 02:54:03.344200 15503 consensus_queue.cc:260] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } } peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } }
I20260806 02:54:03.345446 15503 ts_tablet_manager.cc:1428] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260806 02:54:03.347023 15392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:57883
I20260806 02:54:03.349658 15651 heartbeater.cc:503] Master 127.24.130.190:41015 was elected leader, sending a full tablet report...
I20260806 02:54:03.353555 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.346638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.24.130.129:37641)
0806 02:54:03.347111 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.347126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.348075 (+ 949us) server_negotiation.cc:407] Connection header received
0806 02:54:03.348282 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.348288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.348766 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.348926 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:03.349439 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.350545 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.352385 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.352905 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.353008 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.353104 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.353366 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.353442 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":128,"threads_started":1}
I20260806 02:54:03.353608 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.346338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:37641 (local address 127.0.0.1:34624)
0806 02:54:03.347931 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:03.347966 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:03.348123 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:03.349154 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:03.349159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:03.349184 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:03.349310 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.349316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.350696 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.350700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.352245 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.352261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.353034 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.353038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.353166 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.353208 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 02:54:03.353325 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:54:03.353459 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"thread_start_us":106,"threads_started":1}
I20260806 02:54:03.354949 15557 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d1c6a45caf4470a8b10409dd52e07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:42 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:34624
I20260806 02:54:03.355830 15664 tablet_replica.cc:330] stopping tablet replica
I20260806 02:54:03.356164 15664 raft_consensus.cc:2229] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:03.356443 15664 raft_consensus.cc:2258] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:03.357475 15664 ts_tablet_manager.cc:1905] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:03.361585 15664 ts_tablet_manager.cc:1918] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:03.361675 15664 log.cc:1196] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal/wals/10d1c6a45caf4470a8b10409dd52e07b
I20260806 02:54:03.361991 15664 ts_tablet_manager.cc:1939] T 10d1c6a45caf4470a8b10409dd52e07b P 8f8239525993478c993f8414bf61c25f: Deleting consensus metadata
I20260806 02:54:03.363854 15369 catalog_manager.cc:4841] TS 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641): tablet 10d1c6a45caf4470a8b10409dd52e07b (table test-table [id=0c8a189e2daa4d638ccfed7804bd0409]) successfully deleted
W20260806 02:54:03.456918 15658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:03.457257 15658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:03.457337 15658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:03.461585 15658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:03.461700 15658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:03.461802 15658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:03.467437 15658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:37033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=45253
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:03.468612 15658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:03.469762 15658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:03.479902 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.479912 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.479904 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:03.480367 15658 server_base.cc:1034] running on GCE node
I20260806 02:54:03.480846 15658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:03.481561 15658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:03.482740 15658 hybrid_clock.cc:648] HybridClock initialized: now 1785984843482734 us; error 34 us; skew 500 ppm
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:03.485635 15658 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:54:03.486632 15658 webserver.cc:460] Webserver started at http://127.24.130.130:45253/ using document root <none> and password file <none>
I20260806 02:54:03.487308 15658 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:03.487435 15658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:03.491628 15658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:54:03.493923 15682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.495263 15658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:54:03.495424 15658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "bc4bbac4292f1deb78d96f983f61032e"
server_key_iv: "7d2b4c8b4ada5bf04e134bc212353b33"
server_key_version: "encryptionkey@0"
I20260806 02:54:03.496027 15658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:03.514791 15658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:03.518121 15658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:03.518373 15658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:03.519074 15658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:03.520313 15689 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 02:54:03.524345 15689 ts_tablet_manager.cc:1852] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 02:54:03.524777 15689 ts_tablet_manager.cc:1905] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:03.529021 15689 ts_tablet_manager.cc:1918] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:03.529132 15689 log.cc:1196] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/10d1c6a45caf4470a8b10409dd52e07b
I20260806 02:54:03.529438 15689 ts_tablet_manager.cc:1939] T 10d1c6a45caf4470a8b10409dd52e07b P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting consensus metadata
I20260806 02:54:03.529723 15658 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 02:54:03.529824 15658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260806 02:54:03.530025 15658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:03.530100 15658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.539583 15658 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37033
I20260806 02:54:03.539613 15839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37033 every 8 connection(s)
I20260806 02:54:03.540544 15658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:03.550890 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 15658
I20260806 02:54:03.551520 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:45193
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:41015
--webserver_port=42307
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:54:03.556371 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.542619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:53367 (local address 127.24.130.190:41015)
0806 02:54:03.542837 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.542841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.543725 (+ 884us) server_negotiation.cc:407] Connection header received
0806 02:54:03.544734 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.544738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.544805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.544917 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:03.546147 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.546778 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.547967 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.548222 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.548350 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.551502 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:03.551508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:03.551511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:03.551544 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:03.553722 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.554467 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.554473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.554475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.554589 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.555099 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.555102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.555103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.555307 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:03.555394 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.556113 (+ 719us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.556241 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 02:54:03.557701 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.543021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:53367)
0806 02:54:03.543538 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:03.543585 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:03.544500 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:03.545103 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:03.545110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:03.545472 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:03.545972 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.545987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.546915 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.546923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.547821 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.547832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.548400 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.548404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.548582 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.549244 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:03.549260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:03.551239 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:03.553873 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:03.553881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:03.553902 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:03.554319 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:03.554747 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:03.554749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:03.554751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:03.554980 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:03.555432 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:03.555442 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:03.555819 (+ 377us) client_negotiation.cc:769] Sending connection context
0806 02:54:03.556042 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 02:54:03.556443 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
I20260806 02:54:03.559203 15840 heartbeater.cc:347] Connected to a master server at 127.24.130.190:41015
I20260806 02:54:03.559491 15840 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:03.560060 15840 heartbeater.cc:511] Master 127.24.130.190:41015 requested a full tablet report, sending...
I20260806 02:54:03.561278 15392 ts_manager.cc:194] Registered new tserver with Master: dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033)
I20260806 02:54:03.561921 15392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:53367
W20260806 02:54:03.672339 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:03.672645 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:03.672744 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:03.677037 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:03.677119 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:03.677210 15843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:03.682560 15843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:45193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=42307
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:03.683660 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:03.684700 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:03.694554 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.694548 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:03.694562 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:03.694916 15843 server_base.cc:1034] running on GCE node
I20260806 02:54:03.695327 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:03.695981 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:03.697149 15843 hybrid_clock.cc:648] HybridClock initialized: now 1785984843697128 us; error 34 us; skew 500 ppm
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:03.700217 15843 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:54:03.701148 15843 webserver.cc:460] Webserver started at http://127.24.130.131:42307/ using document root <none> and password file <none>
I20260806 02:54:03.701713 15843 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:03.701834 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:03.705794 15843 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:54:03.707921 15862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.709179 15843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:54:03.709335 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "008867d79c6d475e82497472ef7ad932"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "6fd99e72b869462cb6293fbd053176ea"
server_key_iv: "5e9c31e2f090194cdaa03d93c0dad15e"
server_key_version: "encryptionkey@0"
I20260806 02:54:03.709921 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:03.737722 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:03.747123 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:03.747354 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:03.748021 15843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:03.748953 15843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:03.749028 15843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.749099 15843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:03.749148 15843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:03.759032 15843 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:45193
I20260806 02:54:03.759069 16014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:45193 every 8 connection(s)
I20260806 02:54:03.760149 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:03.761226 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 15843
Aug 06 02:54:03 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984843, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:03.771373 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.762129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.24.130.190:41015)
0806 02:54:03.762288 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.762293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.762602 (+ 309us) server_negotiation.cc:407] Connection header received
0806 02:54:03.762767 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.762769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.762816 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.762919 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:03.763754 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.764314 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.765373 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.765661 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.765781 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.767534 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:03.767538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:03.767540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:03.767564 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:03.769397 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.769892 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.769894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.769896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.769975 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.770310 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.770313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.770314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.770549 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:03.770783 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.771025 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.771164 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:54:03.774921 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.762242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:53967 (local address 127.24.130.190:41015)
0806 02:54:03.762897 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.762904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.763778 (+ 874us) server_negotiation.cc:407] Connection header received
0806 02:54:03.764703 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.764706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.764759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.764856 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:03.766074 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.766800 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.768024 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.768246 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.768326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.771197 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:03.771201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:03.771203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:03.771230 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:03.772906 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.773504 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.773507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.773508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.773588 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.773906 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.773908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.773909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.774069 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:03.774145 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.774629 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.774794 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":91,"threads_started":1}
I20260806 02:54:03.775826 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.762847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:53967)
0806 02:54:03.763638 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:03.763694 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:03.764501 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:03.765015 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:03.765022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:03.765381 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:03.765912 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.765934 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.766917 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.766921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.767901 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.767914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.768386 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.768389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.768533 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.769150 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:03.769166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:03.771009 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:03.773033 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:03.773039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:03.773054 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:03.773410 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:03.773673 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:03.773675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:03.773676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:03.773820 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:03.774157 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:03.774162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:03.774412 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 02:54:03.774578 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:54:03.774921 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":154,"threads_started":1}
I20260806 02:54:03.776989 16016 heartbeater.cc:347] Connected to a master server at 127.24.130.190:41015
I20260806 02:54:03.777288 16016 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:03.777904 16016 heartbeater.cc:511] Master 127.24.130.190:41015 requested a full tablet report, sending...
I20260806 02:54:03.779042 15392 ts_manager.cc:194] Registered new tserver with Master: 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193)
I20260806 02:54:03.779618 15392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:53967
I20260806 02:54:03.784994 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:03.788957 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.785632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.24.130.190:41015)
0806 02:54:03.785765 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.785768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.785953 (+ 185us) server_negotiation.cc:407] Connection header received
0806 02:54:03.786094 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.786096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.786192 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.786277 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:03.786469 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.787036 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.787889 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.788104 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.788179 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.788520 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:03.788650 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 02:54:03.788860 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39170: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:54:03.789042 16020 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39170: token expired
W20260806 02:54:03.789446 15288 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.130.190:41015)
I20260806 02:54:03.797722 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.789865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.24.130.190:41015)
0806 02:54:03.789988 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.789995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.790065 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:54:03.790211 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.790214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.790258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.790362 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:03.791080 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.791681 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.792502 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.792714 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.792824 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.794238 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:03.794246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:03.794249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:03.794272 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:03.795871 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.796387 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.796391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.796395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.796481 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:03.796773 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:03.796778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:03.796780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:03.797014 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:03.797158 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.797423 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.797546 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:54:03.800592 15392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:03.803961 15392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:03.812265 15557 tablet_service.cc:1467] Processing CreateTablet for tablet 270e5983db994876b6101a4f30fa7995 (DEFAULT_TABLE table=test-table [id=003fb1b5956e4018815dd244a4daf305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:03.813019 15557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e5983db994876b6101a4f30fa7995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:03.816084 15503 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Bootstrap starting.
I20260806 02:54:03.816758 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.811079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37033 (local address 127.0.0.1:57770)
0806 02:54:03.811250 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:03.811270 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:03.811359 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:03.812438 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:03.812441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:03.812464 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:03.812575 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.812581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.814195 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.814199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.815313 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.815325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.816358 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.816361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.816446 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.816468 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:54:03.816524 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 02:54:03.816613 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":4}
I20260806 02:54:03.817109 15503 tablet_bootstrap.cc:654] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:03.817462 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.811332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.24.130.130:37033)
0806 02:54:03.811876 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.811890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.811919 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:54:03.812007 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.812012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.812266 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.812450 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:03.812726 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.814057 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.815449 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.816208 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.816980 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.817111 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.817168 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.817278 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":158,"threads_started":1}
I20260806 02:54:03.817745 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.811497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.24.130.131:45193)
0806 02:54:03.812095 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:03.812105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:03.812124 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:54:03.812200 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:03.812204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:03.812358 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:03.812487 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:03.812962 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.813901 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.815256 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.816130 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.816359 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.816539 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:03.817460 (+ 921us) server_negotiation.cc:299] Negotiation successful
0806 02:54:03.817572 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":173,"threads_started":1}
I20260806 02:54:03.817966 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:03.811223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:45193 (local address 127.0.0.1:44606)
0806 02:54:03.811693 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:03.811706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:03.811812 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:03.812659 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:03.812664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:03.812685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:03.812792 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.812798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.814101 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.814104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.815122 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:03.815133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:03.816601 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:03.816604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:03.817138 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:03.817334 (+ 196us) client_negotiation.cc:769] Sending connection context
0806 02:54:03.817421 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 02:54:03.817506 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":166,"threads_started":1}
I20260806 02:54:03.818531 15503 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: No bootstrap required, opened a new log
I20260806 02:54:03.818641 15503 ts_tablet_manager.cc:1397] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 02:54:03.819445 15503 raft_consensus.cc:348] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:03.819583 15503 raft_consensus.cc:374] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:03.819638 15503 raft_consensus.cc:729] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8239525993478c993f8414bf61c25f, State: Initialized, Role: FOLLOWER
I20260806 02:54:03.819229 15747 tablet_service.cc:1467] Processing CreateTablet for tablet 270e5983db994876b6101a4f30fa7995 (DEFAULT_TABLE table=test-table [id=003fb1b5956e4018815dd244a4daf305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:03.819787 15503 consensus_queue.cc:260] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:03.820068 15503 ts_tablet_manager.cc:1428] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260806 02:54:03.820393 15652 tablet.cc:2369] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:03.820439 15747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e5983db994876b6101a4f30fa7995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:03.821568 15925 tablet_service.cc:1467] Processing CreateTablet for tablet 270e5983db994876b6101a4f30fa7995 (DEFAULT_TABLE table=test-table [id=003fb1b5956e4018815dd244a4daf305]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:03.822640 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e5983db994876b6101a4f30fa7995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:03.826298 15689 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Bootstrap starting.
I20260806 02:54:03.829078 15689 tablet_bootstrap.cc:654] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:03.830189 15689 log.cc:824] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:03.832492 16035 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Bootstrap starting.
I20260806 02:54:03.833067 15689 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: No bootstrap required, opened a new log
I20260806 02:54:03.833315 15689 ts_tablet_manager.cc:1397] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:54:03.835116 16035 tablet_bootstrap.cc:654] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:03.836213 16035 log.cc:824] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:03.838636 16035 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: No bootstrap required, opened a new log
I20260806 02:54:03.838865 16035 ts_tablet_manager.cc:1397] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:54:03.838685 15689 raft_consensus.cc:348] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:03.839085 15689 raft_consensus.cc:374] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:03.839164 15689 raft_consensus.cc:729] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4af35ab0014bc3b0fbc67947c2bb03, State: Initialized, Role: FOLLOWER
I20260806 02:54:03.839876 15689 consensus_queue.cc:260] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:03.842085 15840 heartbeater.cc:503] Master 127.24.130.190:41015 was elected leader, sending a full tablet report...
I20260806 02:54:03.843451 16035 raft_consensus.cc:348] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:03.843772 16035 raft_consensus.cc:374] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:03.843848 16035 raft_consensus.cc:729] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008867d79c6d475e82497472ef7ad932, State: Initialized, Role: FOLLOWER
I20260806 02:54:03.844103 15689 ts_tablet_manager.cc:1428] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 02:54:03.844349 16035 consensus_queue.cc:260] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:03.845170 16016 heartbeater.cc:503] Master 127.24.130.190:41015 was elected leader, sending a full tablet report...
I20260806 02:54:03.845443 16035 ts_tablet_manager.cc:1428] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
W20260806 02:54:04.011272 16017 tablet.cc:2369] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:54:04.041890 15841 tablet.cc:2369] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:04.102921 16038 raft_consensus.cc:482] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:04.103183 16038 raft_consensus.cc:504] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:04.105158 16038 leader_election.cc:288] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033), 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641)
I20260806 02:54:04.110054 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:04.105109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37033 (local address 127.24.130.131:41195)
0806 02:54:04.105434 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:04.105450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:04.105551 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:04.106460 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:04.106463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:04.106486 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:04.106671 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:04.106680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.107677 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.107680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:04.108914 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:04.108924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.109590 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.109593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:04.109712 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:04.109738 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:54:04.109830 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:54:04.109908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260806 02:54:04.110054 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:04.106028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:41195 (local address 127.24.130.130:37033)
0806 02:54:04.106198 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:04.106202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:04.106219 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:54:04.106281 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:04.106284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:04.106344 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:04.106410 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:04.106841 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.107531 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.109044 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.109457 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.109598 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:04.109659 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:04.109851 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 02:54:04.109927 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:54:04.111106 15772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e5983db994876b6101a4f30fa7995" candidate_uuid: "008867d79c6d475e82497472ef7ad932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" is_pre_election: true
I20260806 02:54:04.111420 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:04.105123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:37641 (local address 127.24.130.131:38385)
0806 02:54:04.106025 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:04.106038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:04.106175 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:04.107560 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:04.107563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:04.107585 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:04.107728 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:04.107734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.108770 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.108773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:04.110017 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:04.110027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.110975 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.110979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:04.111071 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:04.111097 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:54:04.111196 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:54:04.111271 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":515,"thread_start_us":171,"threads_started":1}
I20260806 02:54:04.111527 15772 raft_consensus.cc:2454] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 008867d79c6d475e82497472ef7ad932 in term 0.
I20260806 02:54:04.111449 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:04.106996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:38385 (local address 127.24.130.129:37641)
0806 02:54:04.107295 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:04.107300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:04.107315 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:54:04.107369 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:04.107372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:04.107417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:04.107521 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:04.107859 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.108637 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.110167 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.110835 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.110949 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:04.110997 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:04.111216 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 02:54:04.111308 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
I20260806 02:54:04.112061 15582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e5983db994876b6101a4f30fa7995" candidate_uuid: "008867d79c6d475e82497472ef7ad932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8239525993478c993f8414bf61c25f" is_pre_election: true
I20260806 02:54:04.112090 15867 leader_election.cc:302] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 008867d79c6d475e82497472ef7ad932, dc4af35ab0014bc3b0fbc67947c2bb03; no voters:
I20260806 02:54:04.112353 15582 raft_consensus.cc:2454] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 008867d79c6d475e82497472ef7ad932 in term 0.
I20260806 02:54:04.112411 16038 raft_consensus.cc:2781] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:04.112489 16038 raft_consensus.cc:482] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:04.112521 16038 raft_consensus.cc:3037] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:04.113781 16038 raft_consensus.cc:504] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:04.114210 16038 leader_election.cc:288] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [CANDIDATE]: Term 1 election: Requested vote from peers dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033), 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641)
I20260806 02:54:04.114567 15582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e5983db994876b6101a4f30fa7995" candidate_uuid: "008867d79c6d475e82497472ef7ad932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8239525993478c993f8414bf61c25f"
I20260806 02:54:04.114614 15772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e5983db994876b6101a4f30fa7995" candidate_uuid: "008867d79c6d475e82497472ef7ad932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
I20260806 02:54:04.114737 15582 raft_consensus.cc:3037] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:04.114784 15772 raft_consensus.cc:3037] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:04.115806 15772 raft_consensus.cc:2454] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 008867d79c6d475e82497472ef7ad932 in term 1.
I20260806 02:54:04.116020 15582 raft_consensus.cc:2454] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 008867d79c6d475e82497472ef7ad932 in term 1.
I20260806 02:54:04.116168 15867 leader_election.cc:302] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 008867d79c6d475e82497472ef7ad932, dc4af35ab0014bc3b0fbc67947c2bb03; no voters:
I20260806 02:54:04.116461 16038 raft_consensus.cc:2781] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:04.116684 16038 raft_consensus.cc:686] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 1 LEADER]: Becoming Leader. State: Replica: 008867d79c6d475e82497472ef7ad932, State: Running, Role: LEADER
I20260806 02:54:04.117053 16038 consensus_queue.cc:237] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:04.120600 15392 catalog_manager.cc:5495] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 reported cstate change: term changed from 0 to 1, leader changed from <none> to 008867d79c6d475e82497472ef7ad932 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "008867d79c6d475e82497472ef7ad932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } health_report { overall_health: UNKNOWN } } }
I20260806 02:54:04.197896 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:04.194242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.24.130.131:45193)
0806 02:54:04.194383 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:04.194386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:04.194407 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:54:04.194637 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:04.194640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:04.194740 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:04.194845 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:04.195055 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.195581 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.196461 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:04.196672 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:04.196754 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:04.197045 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:04.197491 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:04.197587 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:04.197718 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 02:54:04.197775 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:54:04.204743 15772 raft_consensus.cc:1261] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Refusing update from remote peer 008867d79c6d475e82497472ef7ad932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:04.204743 15582 raft_consensus.cc:1261] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Refusing update from remote peer 008867d79c6d475e82497472ef7ad932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:04.205710 16038 consensus_queue.cc:1035] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:04.205969 16043 consensus_queue.cc:1035] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:04.213284 16046 mvcc.cc:205] Tried to move back new op lower bound from 7315393921853231104 to 7315393921506762752. Current Snapshot: MvccSnapshot[applied={T|T < 7315393921853231104}]
I20260806 02:54:04.213645 16047 mvcc.cc:205] Tried to move back new op lower bound from 7315393921853231104 to 7315393921506762752. Current Snapshot: MvccSnapshot[applied={T|T < 7315393921853231104}]
I20260806 02:54:04.224205 15392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:04.224429 15392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:04.227258 15392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Sending DeleteTablet for 3 replicas of tablet 270e5983db994876b6101a4f30fa7995
I20260806 02:54:04.228166 15557 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e5983db994876b6101a4f30fa7995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:04 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:34624
I20260806 02:54:04.228169 15747 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e5983db994876b6101a4f30fa7995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:04 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:57770
I20260806 02:54:04.228240 15925 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e5983db994876b6101a4f30fa7995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:04 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:44606
I20260806 02:54:04.228638 16058 tablet_replica.cc:330] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: stopping tablet replica
I20260806 02:54:04.228715 16060 tablet_replica.cc:330] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: stopping tablet replica
I20260806 02:54:04.228878 16058 raft_consensus.cc:2229] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:04.228946 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 15442
I20260806 02:54:04.229059 16060 raft_consensus.cc:2229] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:04.229190 16058 raft_consensus.cc:2258] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:04.229508 16060 raft_consensus.cc:2258] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:04.229909 16059 tablet_replica.cc:330] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: stopping tablet replica
I20260806 02:54:04.230196 16059 raft_consensus.cc:2229] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:04.230468 16059 raft_consensus.cc:2258] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:04.233677 16059 ts_tablet_manager.cc:1905] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:04.237140 16059 ts_tablet_manager.cc:1918] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:04.237237 16059 log.cc:1196] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/270e5983db994876b6101a4f30fa7995
I20260806 02:54:04.237502 16059 ts_tablet_manager.cc:1939] T 270e5983db994876b6101a4f30fa7995 P dc4af35ab0014bc3b0fbc67947c2bb03: Deleting consensus metadata
I20260806 02:54:04.239044 15371 catalog_manager.cc:4841] TS dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033): tablet 270e5983db994876b6101a4f30fa7995 (table test-table [id=003fb1b5956e4018815dd244a4daf305]) successfully deleted
I20260806 02:54:04.244917 16060 ts_tablet_manager.cc:1905] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:04.249356 16060 ts_tablet_manager.cc:1918] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:04.249495 16060 log.cc:1196] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/270e5983db994876b6101a4f30fa7995
I20260806 02:54:04.249854 16060 ts_tablet_manager.cc:1939] T 270e5983db994876b6101a4f30fa7995 P 008867d79c6d475e82497472ef7ad932: Deleting consensus metadata
I20260806 02:54:04.251370 15369 catalog_manager.cc:4841] TS 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193): tablet 270e5983db994876b6101a4f30fa7995 (table test-table [id=003fb1b5956e4018815dd244a4daf305]) successfully deleted
W20260806 02:54:04.258185 15369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:37641 (error 108)
I20260806 02:54:04.258893 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 15658
I20260806 02:54:04.265486 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:04.265021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:37641 (local address 127.0.0.1:34634)
0806 02:54:04.265277 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:04.265373 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:37641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260806 02:54:04.265691 15369 catalog_manager.cc:4584] TS 8f8239525993478c993f8414bf61c25f (127.24.130.129:37641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 270e5983db994876b6101a4f30fa7995: Network error: Client connection negotiation failed: client connection to 127.24.130.129:37641: connect: Connection refused (error 111)
I20260806 02:54:04.280416 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 15843
I20260806 02:54:04.293059 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 15343
I20260806 02:54:24.306257 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41015
--webserver_interface=127.24.130.190
--webserver_port=43915
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:24.425735 16062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:24.426095 16062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:24.426189 16062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:24.430505 16062 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:54:24.430605 16062 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:54:24.430632 16062 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:54:24.430657 16062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:24.430729 16062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:24.430795 16062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:24.430832 16062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:24.430866 16062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:24.436352 16062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=43915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:24.437518 16062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:24.438607 16062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:54:24.448313 16062 server_base.cc:1034] running on GCE node
W20260806 02:54:24.448272 16070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:24.448266 16071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:24.448261 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:24.449234 16062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:24.450450 16062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:24.451628 16062 hybrid_clock.cc:648] HybridClock initialized: now 1785984864451616 us; error 35 us; skew 500 ppm
Aug 06 02:54:24 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:24.454932 16062 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:54:24.455993 16062 webserver.cc:460] Webserver started at http://127.24.130.190:43915/ using document root <none> and password file <none>
I20260806 02:54:24.456602 16062 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:24.456705 16062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:24.460963 16062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:54:24.463317 16083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:24.464672 16062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:24.464823 16062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "23bd1f44f78140eabd9a552c240c450f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "68432a3744bac485276a181fade97079"
server_key_iv: "92c1d4e07ec6f6746a8b38821f0ac804"
server_key_version: "encryptionkey@0"
I20260806 02:54:24.465431 16062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:24.484216 16062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:24.488106 16062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:24.488369 16062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:24.496281 16062 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:41015
I20260806 02:54:24.496635 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:41015 every 8 connection(s)
I20260806 02:54:24.497483 16062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:24.502084 16153 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:54:24.503997 16153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap starting.
I20260806 02:54:24.505461 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 16062
I20260806 02:54:24.505975 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:37641
--local_ip_for_outbound_sockets=127.24.130.129
--tserver_master_addrs=127.24.130.190:41015
--webserver_port=35801
--webserver_interface=127.24.130.129
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:54:24.522480 16153 log.cc:824] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:24.534826 16153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:54:24.535322 16153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Bootstrap complete.
I20260806 02:54:24.539767 16153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:24.540247 16153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bd1f44f78140eabd9a552c240c450f, State: Initialized, Role: FOLLOWER
I20260806 02:54:24.540845 16153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:24.541005 16153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:24.541110 16153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:24.541244 16153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:54:24.543435 16153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:24.543808 16153 leader_election.cc:302] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23bd1f44f78140eabd9a552c240c450f; no voters:
I20260806 02:54:24.544229 16153 leader_election.cc:288] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:54:24.544332 16159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:54:24.544555 16159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [term 3 LEADER]: Becoming Leader. State: Replica: 23bd1f44f78140eabd9a552c240c450f, State: Running, Role: LEADER
I20260806 02:54:24.544879 16159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } }
I20260806 02:54:24.545586 16153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:24.546505 16162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bd1f44f78140eabd9a552c240c450f. Latest consensus state: current_term: 3 leader_uuid: "23bd1f44f78140eabd9a552c240c450f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } } }
I20260806 02:54:24.546516 16161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23bd1f44f78140eabd9a552c240c450f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bd1f44f78140eabd9a552c240c450f" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41015 } } }
I20260806 02:54:24.546694 16162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:24.546698 16161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:24.547235 16173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:24.552781 16173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:24.553675 16173 catalog_manager.cc:1259] Loaded cluster ID: 69f2e9ff4da347cb9cab4e1b1f562050
I20260806 02:54:24.553748 16173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:24.556561 16173 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:24.557394 16173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Loaded TSK: 0
I20260806 02:54:24.557493 16173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23bd1f44f78140eabd9a552c240c450f: Loaded TSK: 1
F20260806 02:54:24.558339 16173 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd1703fd9ad google::LogMessage::Fail() at ??:0
@ 0x7fd17040088a google::LogMessage::SendToLog() at ??:0
@ 0x7fd1703fd4de google::LogMessage::Flush() at ??:0
@ 0x7fd1703fea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd1730c118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd1730b9f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd17ad627bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd17ad3c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd17ad3d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd17ad7edf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd17ad9d82b std::function<>::operator()() at ??:0
@ 0x7fd17ad3cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd17ad3df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd17ad3a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd17ad7e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd17264f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd17264fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd1726513f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd172640512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd173c846db start_thread at ??:0
@ 0x7fd16eefb61f clone at ??:0
W20260806 02:54:24.651887 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:24.652137 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:24.652182 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:24.656647 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:24.656700 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:24.656762 16157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:24.663002 16157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:37641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=35801
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:24.664122 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:24.665120 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:24.675477 16228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:24.675468 16226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:24.675819 16157 server_base.cc:1034] running on GCE node
W20260806 02:54:24.675963 16225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:24.676405 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:24.677078 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:24.678288 16157 hybrid_clock.cc:648] HybridClock initialized: now 1785984864678263 us; error 46 us; skew 500 ppm
Aug 06 02:54:24 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:24.681437 16157 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:54:24.682438 16157 webserver.cc:460] Webserver started at http://127.24.130.129:35801/ using document root <none> and password file <none>
I20260806 02:54:24.683112 16157 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:24.683215 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:24.687482 16157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:24.690028 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:24.691493 16157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:54:24.691640 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "8f8239525993478c993f8414bf61c25f"
format_stamp: "Formatted at 2026-08-06 02:53:41 on dist-test-slave-3kk6"
server_key: "6c8c3e4eceb099a90398fa7c5fa530fc"
server_key_iv: "4f9fb4807c2f0a718837c05e24a69e40"
server_key_version: "encryptionkey@0"
I20260806 02:54:24.692246 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:24.715852 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:24.719528 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:24.719750 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:24.720412 16157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:24.721640 16247 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:54:24.727501 16157 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:54:24.727595 16157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 02:54:24.727703 16157 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:54:24.729534 16157 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:54:24.729612 16157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:54:24.729990 16247 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Bootstrap starting.
I20260806 02:54:24.741130 16157 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:37641
I20260806 02:54:24.741398 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:37641 every 8 connection(s)
I20260806 02:54:24.742244 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:24.749110 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 16157
I20260806 02:54:24.749615 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:37033
--local_ip_for_outbound_sockets=127.24.130.130
--tserver_master_addrs=127.24.130.190:41015
--webserver_port=45253
--webserver_interface=127.24.130.130
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:54:24.752866 16247 log.cc:824] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:24.763756 16247 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:54:24.764319 16247 tablet_bootstrap.cc:492] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Bootstrap complete.
I20260806 02:54:24.765823 16247 ts_tablet_manager.cc:1397] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.007s
I20260806 02:54:24.769280 16247 raft_consensus.cc:348] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:24.769868 16247 raft_consensus.cc:729] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8239525993478c993f8414bf61c25f, State: Initialized, Role: FOLLOWER
I20260806 02:54:24.770406 16247 consensus_queue.cc:260] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:24.771531 16247 ts_tablet_manager.cc:1428] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
W20260806 02:54:24.820863 16400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:54:24.744388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:56635)
0806 02:54:24.744909 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:24.744969 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:24.745829 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:24.820041 (+ 74212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":164,"threads_started":1}
W20260806 02:54:24.821524 16397 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:41015 (0 consecutive failures): Network error: Failed to ping master at 127.24.130.190:41015: Client connection negotiation failed: client connection to 127.24.130.190:41015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:54:24.822346 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:24.822059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:49737)
0806 02:54:24.822194 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:24.822287 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:54:24.823130 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:24.822908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:45403)
0806 02:54:24.823024 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:24.823081 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:54:24.823335 16397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:54:24.876067 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:24.876442 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:24.876531 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:24.880893 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:24.881009 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:24.881111 16402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:24.886742 16402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:37033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=45253
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:24.888020 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:24.889143 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:24.899245 16413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:24.899204 16415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:24.899235 16412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:24.899493 16402 server_base.cc:1034] running on GCE node
I20260806 02:54:24.899923 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:24.900616 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:24.901787 16402 hybrid_clock.cc:648] HybridClock initialized: now 1785984864901772 us; error 26 us; skew 500 ppm
Aug 06 02:54:24 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:24.904699 16402 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:54:24.905593 16402 webserver.cc:460] Webserver started at http://127.24.130.130:45253/ using document root <none> and password file <none>
I20260806 02:54:24.906142 16402 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:24.906208 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:24.910353 16402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 02:54:24.912582 16424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:24.913902 16402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:54:24.914039 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "dc4af35ab0014bc3b0fbc67947c2bb03"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "bc4bbac4292f1deb78d96f983f61032e"
server_key_iv: "7d2b4c8b4ada5bf04e134bc212353b33"
server_key_version: "encryptionkey@0"
I20260806 02:54:24.914639 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:24.937924 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:24.941886 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:24.942135 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:24.942905 16402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:24.943861 16402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:24.943912 16402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:24.943976 16402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:24.944011 16402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:24.954363 16402 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:37033
I20260806 02:54:24.954385 16573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:37033 every 8 connection(s)
I20260806 02:54:24.955652 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:24.959610 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 16402
I20260806 02:54:24.959996 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:24.958066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:34855)
0806 02:54:24.958856 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:24.958982 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":133,"threads_started":1}
I20260806 02:54:24.960196 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:45193
--local_ip_for_outbound_sockets=127.24.130.131
--tserver_master_addrs=127.24.130.190:41015
--webserver_port=42307
--webserver_interface=127.24.130.131
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:24.960896 16575 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:41015 (0 consecutive failures): Network error: Failed to ping master at 127.24.130.190:41015: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
I20260806 02:54:24.961740 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:24.961471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:33433)
0806 02:54:24.961614 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:24.961672 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:54:24.962598 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:24.962366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:48881)
0806 02:54:24.962484 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:24.962534 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:54:24.962839 16575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:54:25.082891 16579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:25.083214 16579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:25.083269 16579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:25.087453 16579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:25.087543 16579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:25.087608 16579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:25.093298 16579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:45193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=42307
--tserver_master_addrs=127.24.130.190:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:25.094465 16579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:25.095695 16579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:25.105968 16588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:25.105978 16585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:25.105978 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:25.106344 16579 server_base.cc:1034] running on GCE node
I20260806 02:54:25.106875 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:25.107573 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:25.108794 16579 hybrid_clock.cc:648] HybridClock initialized: now 1785984865108782 us; error 37 us; skew 500 ppm
Aug 06 02:54:25 dist-test-slave-3kk6 krb5kdc[14594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984865, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:25.111869 16579 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:54:25.112864 16579 webserver.cc:460] Webserver started at http://127.24.130.131:42307/ using document root <none> and password file <none>
I20260806 02:54:25.113467 16579 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:25.113569 16579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:25.117763 16579 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 02:54:25.120221 16596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:25.121608 16579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:54:25.121788 16579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "008867d79c6d475e82497472ef7ad932"
format_stamp: "Formatted at 2026-08-06 02:53:42 on dist-test-slave-3kk6"
server_key: "6fd99e72b869462cb6293fbd053176ea"
server_key_iv: "5e9c31e2f090194cdaa03d93c0dad15e"
server_key_version: "encryptionkey@0"
I20260806 02:54:25.122412 16579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:25.157826 16579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:25.161752 16579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:25.162010 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:25.162770 16579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:25.163726 16579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:25.163812 16579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:25.163892 16579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:25.163947 16579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:25.174033 16579 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:45193
I20260806 02:54:25.174052 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:45193 every 8 connection(s)
I20260806 02:54:25.175174 16579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:25.179350 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:25.177652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:35841)
0806 02:54:25.178378 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:25.178494 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":136,"threads_started":1}
I20260806 02:54:25.179823 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 16579
W20260806 02:54:25.179946 16764 heartbeater.cc:644] Failed to heartbeat to 127.24.130.190:41015 (0 consecutive failures): Network error: Failed to ping master at 127.24.130.190:41015: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
I20260806 02:54:25.180646 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:25.180397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:36695)
0806 02:54:25.180521 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:25.180578 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:54:25.181409 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:25.181201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:39299)
0806 02:54:25.181313 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:25.181359 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:54:25.181617 16764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:54:25.824607 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:25.824078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:40289)
0806 02:54:25.824406 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:25.824480 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260806 02:54:25.964263 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:25.963627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:39313)
0806 02:54:25.963998 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:25.964090 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":89,"threads_started":1}
I20260806 02:54:26.182996 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.182345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:49459)
0806 02:54:26.182777 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:26.182863 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20260806 02:54:26.272984 16774 raft_consensus.cc:482] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:26.273247 16774 raft_consensus.cc:504] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:26.274636 16774 leader_election.cc:288] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033), 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193)
Aug 06 02:54:26 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.131@KRBTEST.COM
Aug 06 02:54:26 dist-test-slave-3kk6 krb5kdc[14594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984864, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.130@KRBTEST.COM
I20260806 02:54:26.294802 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.275315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:40019 (local address 127.24.130.130:37033)
0806 02:54:26.275669 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:26.275696 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:26.275736 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:54:26.276454 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:26.276460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:26.276816 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:26.277136 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:26.278502 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.279590 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.280930 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.281417 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.281606 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:26.289578 (+ 7972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:26.289583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:26.289596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:26.289630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:26.291646 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:26.292487 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:26.292494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:26.292505 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:26.292641 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:26.293050 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:26.293054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:26.293056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:26.293528 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:26.293680 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:26.294281 (+ 601us) server_negotiation.cc:299] Negotiation successful
0806 02:54:26.294561 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260806 02:54:26.294809 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.274939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:37033 (local address 127.24.130.129:40019)
0806 02:54:26.275354 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:26.275370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:26.275492 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:26.277188 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:26.277194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:26.277786 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:26.278307 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:26.278324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.279756 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.279759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:26.280785 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:26.280796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.281561 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.281565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:26.281804 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:26.287158 (+ 5354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:26.287189 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:26.289320 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:26.291820 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:26.291827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:26.291847 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:26.292319 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:26.292760 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:26.292763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:26.292765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:26.292940 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:26.293654 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:26.293661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:26.293943 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 02:54:26.294150 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 02:54:26.294541 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":230,"spinlock_wait_cycles":79232}
I20260806 02:54:26.294819 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.275156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:39223 (local address 127.24.130.131:45193)
0806 02:54:26.275763 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:26.275789 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:26.275832 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:54:26.276482 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:26.276489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:26.276815 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:26.277135 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:26.278503 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.279583 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.280925 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.281417 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.281612 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:26.289527 (+ 7915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:26.289534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:26.289548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:26.289584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:26.291594 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:26.292491 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:26.292497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:26.292501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:26.292638 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:26.293046 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:26.293050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:26.293053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:26.293528 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:26.293680 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:26.294272 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 02:54:26.294592 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":136,"threads_started":1}
I20260806 02:54:26.294867 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.274952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:45193 (local address 127.24.130.129:39223)
0806 02:54:26.275636 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:26.275650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:26.275762 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:26.277187 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:26.277193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:26.277785 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:26.278308 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:26.278324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.279759 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.279762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:26.280783 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:26.280794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:26.281561 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:26.281565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:26.281803 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:26.287194 (+ 5391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:26.287209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:26.289204 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:26.291830 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:26.291832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:26.291845 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:26.292324 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:26.292749 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:26.292753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:26.292755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:26.292938 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:26.293654 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:26.293661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:26.293943 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 02:54:26.294149 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:54:26.294636 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":435,"spinlock_wait_cycles":13440,"thread_start_us":134,"threads_started":1}
W20260806 02:54:26.298080 16243 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
W20260806 02:54:26.298283 16240 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
I20260806 02:54:26.298353 16243 leader_election.cc:302] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f8239525993478c993f8414bf61c25f; no voters: 008867d79c6d475e82497472ef7ad932, dc4af35ab0014bc3b0fbc67947c2bb03
I20260806 02:54:26.298698 16774 raft_consensus.cc:2726] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:54:26.825941 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.825475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:57445)
0806 02:54:26.825766 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:26.825834 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260806 02:54:26.965981 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:26.965438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:40761)
0806 02:54:26.965785 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:26.965867 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260806 02:54:27.184315 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:27.183862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:54177)
0806 02:54:27.184144 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:27.184220 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260806 02:54:27.827490 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:27.827005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:58267)
0806 02:54:27.827308 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:27.827383 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260806 02:54:27.886169 16784 raft_consensus.cc:482] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:27.886365 16784 raft_consensus.cc:504] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:27.886907 16784 leader_election.cc:288] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033), 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193)
W20260806 02:54:27.887866 16240 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
W20260806 02:54:27.887975 16243 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
I20260806 02:54:27.888041 16243 leader_election.cc:302] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f8239525993478c993f8414bf61c25f; no voters: 008867d79c6d475e82497472ef7ad932, dc4af35ab0014bc3b0fbc67947c2bb03
I20260806 02:54:27.888192 16784 raft_consensus.cc:2726] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:54:27.967463 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:27.966976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:58475)
0806 02:54:27.967276 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:27.967340 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
I20260806 02:54:28.185873 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:28.185298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:53705)
0806 02:54:28.185653 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:28.185735 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260806 02:54:28.828970 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:28.828448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:56017)
0806 02:54:28.828762 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:28.828864 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260806 02:54:28.968750 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:28.968319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:36493)
0806 02:54:28.968586 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:28.968647 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260806 02:54:29.187425 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:29.186920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:33531)
0806 02:54:29.187249 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:29.187322 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260806 02:54:29.830909 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:29.830301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:51197)
0806 02:54:29.830723 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:29.830804 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260806 02:54:29.970320 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:29.969759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:54587)
0806 02:54:29.970127 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:29.970210 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
I20260806 02:54:30.189040 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:30.188494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:33607)
0806 02:54:30.188845 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:30.188934 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260806 02:54:30.832495 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:30.831961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:35729)
0806 02:54:30.832305 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:30.832389 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":114,"threads_started":1}
I20260806 02:54:30.972067 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:30.971502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:34331)
0806 02:54:30.971868 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:30.971961 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260806 02:54:31.190593 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:31.190072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:49657)
0806 02:54:31.190400 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:31.190485 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
I20260806 02:54:31.252733 16806 raft_consensus.cc:482] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:31.252915 16806 raft_consensus.cc:504] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:31.253351 16806 leader_election.cc:288] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033), 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193)
W20260806 02:54:31.254426 16240 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
W20260806 02:54:31.254557 16243 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
I20260806 02:54:31.254632 16243 leader_election.cc:302] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f8239525993478c993f8414bf61c25f; no voters: 008867d79c6d475e82497472ef7ad932, dc4af35ab0014bc3b0fbc67947c2bb03
I20260806 02:54:31.254892 16806 raft_consensus.cc:2726] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:54:31.833846 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:31.833329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:39877)
0806 02:54:31.833658 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:31.833734 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260806 02:54:31.973680 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:31.973104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:34495)
0806 02:54:31.973497 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:31.973575 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":99,"threads_started":1}
I20260806 02:54:32.192137 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:32.191615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:51917)
0806 02:54:32.191979 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:32.192037 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":101,"threads_started":1}
I20260806 02:54:32.835371 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:32.834870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:54437)
0806 02:54:32.835189 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:32.835258 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260806 02:54:32.975319 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:32.974773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:40575)
0806 02:54:32.975124 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:32.975201 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":91,"threads_started":1}
I20260806 02:54:33.193532 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:33.192956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:55975)
0806 02:54:33.193257 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:33.193382 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260806 02:54:33.836609 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:33.836199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:58075)
0806 02:54:33.836457 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:33.836511 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260806 02:54:33.976902 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:33.976414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:48971)
0806 02:54:33.976712 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:33.976789 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20260806 02:54:34.194939 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:34.194410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:44119)
0806 02:54:34.194770 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:34.194838 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260806 02:54:34.648375 16819 raft_consensus.cc:482] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:34.648526 16819 raft_consensus.cc:504] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4af35ab0014bc3b0fbc67947c2bb03" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 37033 } } peers { permanent_uuid: "008867d79c6d475e82497472ef7ad932" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 45193 } } peers { permanent_uuid: "8f8239525993478c993f8414bf61c25f" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 37641 } }
I20260806 02:54:34.648921 16819 leader_election.cc:288] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033), 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193)
W20260806 02:54:34.649935 16240 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 008867d79c6d475e82497472ef7ad932 (127.24.130.131:45193): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
W20260806 02:54:34.650074 16243 leader_election.cc:341] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc4af35ab0014bc3b0fbc67947c2bb03 (127.24.130.130:37033): Not found: Tablet not found: 270e5983db994876b6101a4f30fa7995
I20260806 02:54:34.650184 16243 leader_election.cc:302] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f8239525993478c993f8414bf61c25f; no voters: 008867d79c6d475e82497472ef7ad932, dc4af35ab0014bc3b0fbc67947c2bb03
I20260806 02:54:34.650362 16819 raft_consensus.cc:2726] T 270e5983db994876b6101a4f30fa7995 P 8f8239525993478c993f8414bf61c25f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:54:34.838044 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:34.837552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:49065)
0806 02:54:34.837869 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:34.837940 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260806 02:54:34.978427 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:34.977934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:53865)
0806 02:54:34.978233 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:34.978308 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260806 02:54:35.196492 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:35.195853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:40369)
0806 02:54:35.196317 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:35.196393 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":274,"threads_started":1}
I20260806 02:54:35.839428 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:35.838940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:47547)
0806 02:54:35.839231 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:35.839307 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260806 02:54:35.979899 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:35.979335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:59491)
0806 02:54:35.979703 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:35.979780 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260806 02:54:36.197778 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:36.197272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:33125)
0806 02:54:36.197605 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:36.197675 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260806 02:54:36.840790 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:36.840280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:54431)
0806 02:54:36.840617 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:36.840681 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260806 02:54:36.981513 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:36.980949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:47655)
0806 02:54:36.981331 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:36.981407 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":87,"threads_started":1}
I20260806 02:54:37.199250 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:37.198766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:45177)
0806 02:54:37.199077 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:37.199146 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260806 02:54:37.842226 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:37.841744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:48301)
0806 02:54:37.842055 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:37.842122 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20260806 02:54:37.983265 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:37.982561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:33427)
0806 02:54:37.983051 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:37.983124 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
I20260806 02:54:38.200660 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:38.200133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:58307)
0806 02:54:38.200487 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:38.200555 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":146,"threads_started":1}
I20260806 02:54:38.843595 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:38.843128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:47109)
0806 02:54:38.843432 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:38.843492 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
I20260806 02:54:38.984681 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:38.984160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:55969)
0806 02:54:38.984497 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:38.984577 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260806 02:54:39.202042 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:39.201573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:33073)
0806 02:54:39.201874 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:39.201939 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":106,"threads_started":1}
I20260806 02:54:39.845237 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:39.844754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.129:57705)
0806 02:54:39.845065 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:39.845134 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260806 02:54:39.986498 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:39.985911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.130:35703)
0806 02:54:39.986298 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:39.986387 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260806 02:54:40.188063 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 16157
I20260806 02:54:40.202466 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 16402
I20260806 02:54:40.207232 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:40.203062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:41015 (local address 127.24.130.131:53543)
0806 02:54:40.206985 (+ 3923us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:40.207085 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.190:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3781,"thread_start_us":1651,"threads_started":1}
I20260806 02:54:40.215898 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 16579
2026-08-06T02:54:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60098 ms)
[----------] 30 tests from SecurityITest (255928 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:40 dist-test-slave-3kk6 krb5kdc[16848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:40 dist-test-slave-3kk6 krb5kdc[16848](info): set up 2 sockets
Aug 06 02:54:40 dist-test-slave-3kk6 krb5kdc[16848](info): commencing operation
krb5kdc: starting...
W20260806 02:54:41.476388 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:41 dist-test-slave-3kk6 krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:54:41.617702 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:45541
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:45541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:41.739933 16866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:41.740327 16866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:41.740422 16866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:41.744761 16866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:41.744892 16866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:41.744931 16866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:41.744975 16866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:41.745016 16866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:41.750950 16866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:45541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:45541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:41.752269 16866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:41.753414 16866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:41.763013 16874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:41.763016 16873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:41.763017 16879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:41.763167 16866 server_base.cc:1034] running on GCE node
I20260806 02:54:41.764132 16866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:41.765426 16866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:41.766696 16866 hybrid_clock.cc:648] HybridClock initialized: now 1785984881766628 us; error 57 us; skew 500 ppm
Aug 06 02:54:41 dist-test-slave-3kk6 krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984881, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:41.769829 16866 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:54:41.770876 16866 webserver.cc:460] Webserver started at http://127.24.130.190:39023/ using document root <none> and password file <none>
I20260806 02:54:41.771485 16866 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:41.771551 16866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:41.771806 16866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:41.774044 16866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "a366996d33254a2da357e4afebe7ee2b"
format_stamp: "Formatted at 2026-08-06 02:54:41 on dist-test-slave-3kk6"
server_key: "eb58ca97c4aef6df3d1fc11f807696c4"
server_key_iv: "c62c3d0a5910cc40f8c807882c18e020"
server_key_version: "encryptionkey@0"
I20260806 02:54:41.774742 16866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "a366996d33254a2da357e4afebe7ee2b"
format_stamp: "Formatted at 2026-08-06 02:54:41 on dist-test-slave-3kk6"
server_key: "eb58ca97c4aef6df3d1fc11f807696c4"
server_key_iv: "c62c3d0a5910cc40f8c807882c18e020"
server_key_version: "encryptionkey@0"
I20260806 02:54:41.778877 16866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:54:41.781448 16888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:41.782852 16866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:41.782989 16866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "a366996d33254a2da357e4afebe7ee2b"
format_stamp: "Formatted at 2026-08-06 02:54:41 on dist-test-slave-3kk6"
server_key: "eb58ca97c4aef6df3d1fc11f807696c4"
server_key_iv: "c62c3d0a5910cc40f8c807882c18e020"
server_key_version: "encryptionkey@0"
I20260806 02:54:41.783115 16866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:41.797330 16866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:41.801209 16866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:41.801455 16866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:41.809578 16866 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:45541
I20260806 02:54:41.809607 16963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:45541 every 8 connection(s)
I20260806 02:54:41.810775 16866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:41.813727 16964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:41.816581 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 16866
I20260806 02:54:41.816704 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:41.817047 25098 external_mini_cluster.cc:1428] Setting key c172e0bdee84dcf51735eb35aa5cbcee
I20260806 02:54:41.820621 16964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: Bootstrap starting.
I20260806 02:54:41.823681 16964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:41.824585 16964 log.cc:824] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:41.826887 16964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: No bootstrap required, opened a new log
Aug 06 02:54:41 dist-test-slave-3kk6 krb5kdc[16848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:41.831455 16964 raft_consensus.cc:348] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a366996d33254a2da357e4afebe7ee2b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 45541 } }
I20260806 02:54:41.831794 16964 raft_consensus.cc:374] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:41.831863 16964 raft_consensus.cc:729] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a366996d33254a2da357e4afebe7ee2b, State: Initialized, Role: FOLLOWER
I20260806 02:54:41.832445 16964 consensus_queue.cc:260] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a366996d33254a2da357e4afebe7ee2b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 45541 } }
I20260806 02:54:41.832619 16964 raft_consensus.cc:388] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:41.832692 16964 raft_consensus.cc:482] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:41.832799 16964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:41.834167 16964 raft_consensus.cc:504] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a366996d33254a2da357e4afebe7ee2b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 45541 } }
I20260806 02:54:41.834556 16964 leader_election.cc:302] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a366996d33254a2da357e4afebe7ee2b; no voters:
I20260806 02:54:41.834811 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:41.818723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.24.130.190:45541)
0806 02:54:41.819245 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:41.819266 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:41.819319 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:54:41.820208 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:41.820234 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:41.820690 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:41.821103 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:41.821873 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:41.823461 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:41.824445 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:41.825160 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:41.825298 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:41.828190 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:41.828197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:41.828215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:41.828251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:41.831177 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:41.831662 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:41.831685 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:41.831691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:41.831856 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:41.832222 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:41.832225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:41.832227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:41.832810 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:41.833029 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:41.833314 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 02:54:41.833640 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260806 02:54:41.834986 16964 leader_election.cc:288] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:41.835294 16971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:41.835605 16971 raft_consensus.cc:686] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [term 1 LEADER]: Becoming Leader. State: Replica: a366996d33254a2da357e4afebe7ee2b, State: Running, Role: LEADER
I20260806 02:54:41.836004 16971 consensus_queue.cc:237] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a366996d33254a2da357e4afebe7ee2b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 45541 } }
I20260806 02:54:41.836341 16964 sys_catalog.cc:564] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:41.837839 16974 sys_catalog.cc:455] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a366996d33254a2da357e4afebe7ee2b. Latest consensus state: current_term: 1 leader_uuid: "a366996d33254a2da357e4afebe7ee2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a366996d33254a2da357e4afebe7ee2b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 45541 } } }
I20260806 02:54:41.837960 16973 sys_catalog.cc:455] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a366996d33254a2da357e4afebe7ee2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a366996d33254a2da357e4afebe7ee2b" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 45541 } } }
I20260806 02:54:41.838105 16973 sys_catalog.cc:458] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:41.838105 16974 sys_catalog.cc:458] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b [sys.catalog]: This master's current role is: LEADER
W20260806 02:54:41.841312 16992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:54:41.841415 16992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:54:41.841493 16989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:41.842413 16989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:41.848133 16989 catalog_manager.cc:1347] Generated new cluster ID: b381706b73b44e759a9011d2549afa60
I20260806 02:54:41.848336 16989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:41.866225 16989 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:41.867497 16989 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:41.875097 16989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: Generated new TSK 0
I20260806 02:54:41.876010 16989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:54:41.914611 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:45541
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:42.038873 17000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:42.039201 17000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:42.039247 17000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:42.043550 17000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:42.043628 17000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:42.043730 17000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:42.049368 17000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:45541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:42.050583 17000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:42.051776 17000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:42.062299 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:42.062332 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:42.062619 17000 server_base.cc:1034] running on GCE node
W20260806 02:54:42.062299 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:42.063150 17000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:42.063859 17000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:42.065045 17000 hybrid_clock.cc:648] HybridClock initialized: now 1785984882065029 us; error 31 us; skew 500 ppm
Aug 06 02:54:42 dist-test-slave-3kk6 krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:42.068343 17000 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:54:42.069370 17000 webserver.cc:460] Webserver started at http://127.24.130.129:40385/ using document root <none> and password file <none>
I20260806 02:54:42.069944 17000 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:42.070008 17000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:42.070191 17000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:42.072501 17000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "0599c9a53c64494295c4d9601c1a7771"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "e9394e1bfe3178b47bdaba993c9f7db3"
server_key_iv: "259e55c61bc3e24580b4a101b3ff15c1"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.073092 17000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "0599c9a53c64494295c4d9601c1a7771"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "e9394e1bfe3178b47bdaba993c9f7db3"
server_key_iv: "259e55c61bc3e24580b4a101b3ff15c1"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.077248 17000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:54:42.079901 17020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.081249 17000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:54:42.081362 17000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "0599c9a53c64494295c4d9601c1a7771"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "e9394e1bfe3178b47bdaba993c9f7db3"
server_key_iv: "259e55c61bc3e24580b4a101b3ff15c1"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.081461 17000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:42.095299 17000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:42.099413 17000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:42.099655 17000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:42.100416 17000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:42.101703 17000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:42.101826 17000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 02:54:42.101903 17000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:42.101948 17000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.112643 17000 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:43505
I20260806 02:54:42.112668 17187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:43505 every 8 connection(s)
I20260806 02:54:42.113745 17000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:42.123636 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 17000
I20260806 02:54:42.123761 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:42.124138 25098 external_mini_cluster.cc:1428] Setting key c3136431d41b529e51f090b316b55799
Aug 06 02:54:42 dist-test-slave-3kk6 krb5kdc[16848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:42.129563 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.115882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:34371 (local address 127.24.130.190:45541)
0806 02:54:42.116030 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.116033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.116877 (+ 844us) server_negotiation.cc:407] Connection header received
0806 02:54:42.118323 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.118326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.118392 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.118508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:42.119918 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.120476 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.121721 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.121976 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.122094 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.125230 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:42.125235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:42.125238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:42.125265 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:42.127253 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.127852 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.127856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.127857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.127960 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.128347 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.128350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.128352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.128559 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:42.128675 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.129233 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.129386 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:54:42.130472 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.116211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:45541 (local address 127.24.130.129:34371)
0806 02:54:42.116705 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.116755 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.118077 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.118770 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.118778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.119184 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:42.119723 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.119738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.120626 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.120630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.121579 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.121590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.122160 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.122164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.122351 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.123081 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:42.123098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:42.125040 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:42.127400 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:42.127407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:42.127426 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:42.127732 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:42.128062 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:42.128066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:42.128068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:42.128238 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:42.128676 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:42.128686 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:42.128980 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.129190 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.129582 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260806 02:54:42.131916 17191 heartbeater.cc:347] Connected to a master server at 127.24.130.190:45541
I20260806 02:54:42.132200 17191 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:42.132735 17191 heartbeater.cc:511] Master 127.24.130.190:45541 requested a full tablet report, sending...
I20260806 02:54:42.134717 16916 ts_manager.cc:194] Registered new tserver with Master: 0599c9a53c64494295c4d9601c1a7771 (127.24.130.129:43505)
I20260806 02:54:42.136601 16916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:34371
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:54:42.160526 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:45541
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:42.285110 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:42.285432 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:42.285480 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:42.290045 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:42.290138 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:42.290203 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:42.296105 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:45541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:42.297356 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:42.298440 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:42.309278 17209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:42.309463 17197 server_base.cc:1034] running on GCE node
W20260806 02:54:42.309234 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:42.309234 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:42.310060 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:42.310850 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:42.312026 17197 hybrid_clock.cc:648] HybridClock initialized: now 1785984882312011 us; error 34 us; skew 500 ppm
Aug 06 02:54:42 dist-test-slave-3kk6 krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:42.315392 17197 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:54:42.316443 17197 webserver.cc:460] Webserver started at http://127.24.130.130:46195/ using document root <none> and password file <none>
I20260806 02:54:42.317088 17197 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:42.317159 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:42.317447 17197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:42.319881 17197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "fe75c7cf1b054cceabc6308dfdf4698a"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "ce44994b96982971209bbd4792a37ad8"
server_key_iv: "7dbfe96406e159fed04e9bccc7fc1182"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.320535 17197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "fe75c7cf1b054cceabc6308dfdf4698a"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "ce44994b96982971209bbd4792a37ad8"
server_key_iv: "7dbfe96406e159fed04e9bccc7fc1182"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.324899 17197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:54:42.327812 17218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.329229 17197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:42.329377 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "fe75c7cf1b054cceabc6308dfdf4698a"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "ce44994b96982971209bbd4792a37ad8"
server_key_iv: "7dbfe96406e159fed04e9bccc7fc1182"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.329515 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:42.356268 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:42.359977 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:42.360224 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:42.360939 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:42.361924 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:42.361977 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.362047 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:42.362079 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.371873 17197 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:32991
I20260806 02:54:42.371901 17372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:32991 every 8 connection(s)
I20260806 02:54:42.373080 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:42.379865 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 17197
I20260806 02:54:42.380033 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:42.380390 25098 external_mini_cluster.cc:1428] Setting key e46eb361bcb2035b0ab1976db88950f2
Aug 06 02:54:42 dist-test-slave-3kk6 krb5kdc[16848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:42.389281 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.375447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:50815 (local address 127.24.130.190:45541)
0806 02:54:42.375590 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.375594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.376493 (+ 899us) server_negotiation.cc:407] Connection header received
0806 02:54:42.377825 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.377828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.377876 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.377975 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:42.379578 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.380210 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.381730 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.381965 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.382047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.385188 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:42.385192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:42.385194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:42.385220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:42.387006 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.387592 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.387596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.387597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.387677 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.388054 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.388057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.388059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.388268 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:42.388384 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.388962 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.389106 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:54:42.390167 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.375860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:45541 (local address 127.24.130.130:50815)
0806 02:54:42.376322 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.376368 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.377607 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.378147 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.378153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.378895 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:42.379428 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.379443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.380353 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.380356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.381590 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.381601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.382111 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.382116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.382340 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.383060 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:42.383089 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:42.384928 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:42.387143 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:42.387149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:42.387168 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:42.387467 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:42.387779 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:42.387782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:42.387784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:42.387946 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:42.388385 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:42.388403 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:42.388693 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.388895 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.389268 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260806 02:54:42.391527 17373 heartbeater.cc:347] Connected to a master server at 127.24.130.190:45541
I20260806 02:54:42.391852 17373 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:54:42.392560 17373 heartbeater.cc:511] Master 127.24.130.190:45541 requested a full tablet report, sending...
I20260806 02:54:42.393778 16916 ts_manager.cc:194] Registered new tserver with Master: fe75c7cf1b054cceabc6308dfdf4698a (127.24.130.130:32991)
I20260806 02:54:42.394579 16916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:50815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:54:42.414136 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:45541
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:42.537106 17379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:42.537588 17379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:42.537706 17379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:42.542640 17379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:42.542840 17379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:42.542946 17379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:42.549160 17379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:45541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:42.550411 17379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:42.551558 17379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:42.562374 17385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:42.562371 17387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:42.562371 17384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:42.562973 17379 server_base.cc:1034] running on GCE node
I20260806 02:54:42.563642 17379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:42.564452 17379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:42.565680 17379 hybrid_clock.cc:648] HybridClock initialized: now 1785984882565653 us; error 63 us; skew 500 ppm
Aug 06 02:54:42 dist-test-slave-3kk6 krb5kdc[16848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:42.569010 17379 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:54:42.570171 17379 webserver.cc:460] Webserver started at http://127.24.130.131:35107/ using document root <none> and password file <none>
I20260806 02:54:42.570900 17379 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:42.571015 17379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:42.571277 17379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:42.573817 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "a37c121e43364e6e83616aef54569ea5"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "437eabb17ca96aff09113f8b2049d12d"
server_key_iv: "3893e00e1a922dc47300c3c0add60c53"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.574507 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "a37c121e43364e6e83616aef54569ea5"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "437eabb17ca96aff09113f8b2049d12d"
server_key_iv: "3893e00e1a922dc47300c3c0add60c53"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.578917 17379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:54:42.581799 17396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.583281 17379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:42.583460 17379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "a37c121e43364e6e83616aef54569ea5"
format_stamp: "Formatted at 2026-08-06 02:54:42 on dist-test-slave-3kk6"
server_key: "437eabb17ca96aff09113f8b2049d12d"
server_key_iv: "3893e00e1a922dc47300c3c0add60c53"
server_key_version: "encryptionkey@0"
I20260806 02:54:42.583606 17379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:42.607745 17379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:42.611644 17379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:42.611928 17379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:42.612663 17379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:42.613704 17379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:42.613798 17379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.613912 17379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:42.613968 17379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:42.623937 17379 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:38559
I20260806 02:54:42.623965 17554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:38559 every 8 connection(s)
I20260806 02:54:42.624970 17379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:42.633183 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 17379
I20260806 02:54:42.633340 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:42.633695 25098 external_mini_cluster.cc:1428] Setting key 6954819b568340d5233b15a10a63fb07
Aug 06 02:54:42 dist-test-slave-3kk6 krb5kdc[16848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984882, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:42.640254 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.627053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:47577 (local address 127.24.130.190:45541)
0806 02:54:42.627217 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.627221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.628090 (+ 869us) server_negotiation.cc:407] Connection header received
0806 02:54:42.629503 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.629506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.629572 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.629680 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:42.631201 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.631784 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.633029 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.633280 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.633442 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.636598 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:42.636601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:42.636603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:42.636631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:42.638109 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.638644 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.638667 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.638669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.638766 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.639147 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.639150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.639152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.639365 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:42.639458 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.640005 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.640125 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:54:42.641155 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.627429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:45541 (local address 127.24.130.131:47577)
0806 02:54:42.627923 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.627969 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.629266 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.629880 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.629887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.630275 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:42.631023 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.631045 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.631932 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.631936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.632883 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.632893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.633490 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.633495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.633680 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.634378 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:42.634396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:42.636438 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:42.638226 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:42.638232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:42.638247 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:42.638542 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:42.638868 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:42.638870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:42.638872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:42.639025 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:42.639475 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:42.639481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:42.639751 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.639938 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.640243 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260806 02:54:42.642391 17555 heartbeater.cc:347] Connected to a master server at 127.24.130.190:45541
I20260806 02:54:42.642712 17555 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:42.643275 17555 heartbeater.cc:511] Master 127.24.130.190:45541 requested a full tablet report, sending...
I20260806 02:54:42.644393 16916 ts_manager.cc:194] Registered new tserver with Master: a37c121e43364e6e83616aef54569ea5 (127.24.130.131:38559)
I20260806 02:54:42.644953 16916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:47577
I20260806 02:54:42.648360 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:42.658456 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.650473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.24.130.190:45541)
0806 02:54:42.650636 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.650642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.650944 (+ 302us) server_negotiation.cc:407] Connection header received
0806 02:54:42.651038 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.651040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.651082 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.651157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:42.651921 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.652453 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.653305 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.653523 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.653615 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.655230 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:42.655233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:42.655235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:42.655261 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:42.656746 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.657181 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.657184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.657185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.657264 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:42.657539 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:42.657542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:42.657543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:42.657779 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:42.657896 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.658123 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.658232 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:54:42.661902 16916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:42.665215 16916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:42.680989 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.675568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43505 (local address 127.0.0.1:50660)
0806 02:54:42.675808 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.675823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.675909 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.676795 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.676798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.676817 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:42.676907 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.676913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.678253 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.678257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.679224 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.679262 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.680287 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.680290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.680510 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.680538 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.680639 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.680820 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":154,"thread_start_us":109,"threads_started":1}
I20260806 02:54:42.681172 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.675194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:38559 (local address 127.0.0.1:43068)
0806 02:54:42.675472 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.675489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.675582 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.676586 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.676590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.676610 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:42.676743 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.676756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.678494 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.678498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.679661 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.679672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.680826 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.680829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.680915 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.680936 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.680984 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.681059 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260806 02:54:42.680968 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.675908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.24.130.129:43505)
0806 02:54:42.676310 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.676316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.676339 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:54:42.676413 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.676421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.676613 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.676814 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:42.677008 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.678110 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.679388 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.680130 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.680479 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.680617 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.680673 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.680784 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260806 02:54:42.681485 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.674538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.24.130.130:32991)
0806 02:54:42.675420 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.675436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.675472 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:54:42.675558 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.675563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.675792 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.675964 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:42.676515 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.677521 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.679310 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.680089 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.680229 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.680376 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.680765 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.680858 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":268,"threads_started":1}
I20260806 02:54:42.681972 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.674378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:32991 (local address 127.0.0.1:44542)
0806 02:54:42.674806 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.674871 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.675013 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.676180 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.676185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.676210 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:42.676374 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.676382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.677703 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.677707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.679164 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.679176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.680231 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.680238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.680462 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.680523 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.680649 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.681815 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":1092,"thread_start_us":101,"threads_started":1}
I20260806 02:54:42.682430 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.675783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.24.130.131:38559)
0806 02:54:42.676179 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.676199 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.676226 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:54:42.676297 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.676301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.676460 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.676625 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:42.676890 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.678319 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.679816 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.680680 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.682030 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.682154 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.682206 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.682296 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":187,"threads_started":1}
I20260806 02:54:42.684882 17092 tablet_service.cc:1467] Processing CreateTablet for tablet 9007b48532664fd5ae99abcea69858ef (DEFAULT_TABLE table=test-table [id=916eb7b12acb417dad9e17d182d1326d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:42.684878 17282 tablet_service.cc:1467] Processing CreateTablet for tablet 9007b48532664fd5ae99abcea69858ef (DEFAULT_TABLE table=test-table [id=916eb7b12acb417dad9e17d182d1326d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:42.685331 17465 tablet_service.cc:1467] Processing CreateTablet for tablet 9007b48532664fd5ae99abcea69858ef (DEFAULT_TABLE table=test-table [id=916eb7b12acb417dad9e17d182d1326d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:42.686172 17092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9007b48532664fd5ae99abcea69858ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:42.686175 17282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9007b48532664fd5ae99abcea69858ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:42.686228 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9007b48532664fd5ae99abcea69858ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:42.694527 17577 tablet_bootstrap.cc:492] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Bootstrap starting.
I20260806 02:54:42.695067 17576 tablet_bootstrap.cc:492] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: Bootstrap starting.
I20260806 02:54:42.695520 17579 tablet_bootstrap.cc:492] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Bootstrap starting.
I20260806 02:54:42.697999 17577 tablet_bootstrap.cc:654] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:42.698011 17579 tablet_bootstrap.cc:654] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:42.698011 17576 tablet_bootstrap.cc:654] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:42.699113 17576 log.cc:824] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:42.699327 17579 log.cc:824] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:42.699436 17577 log.cc:824] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:42.701321 17579 tablet_bootstrap.cc:492] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: No bootstrap required, opened a new log
I20260806 02:54:42.701334 17576 tablet_bootstrap.cc:492] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: No bootstrap required, opened a new log
I20260806 02:54:42.701548 17576 ts_tablet_manager.cc:1397] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:54:42.701550 17579 ts_tablet_manager.cc:1397] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260806 02:54:42.703135 17577 tablet_bootstrap.cc:492] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: No bootstrap required, opened a new log
I20260806 02:54:42.703349 17577 ts_tablet_manager.cc:1397] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 02:54:42.705788 17579 raft_consensus.cc:348] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.706103 17579 raft_consensus.cc:374] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:42.706156 17579 raft_consensus.cc:729] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe75c7cf1b054cceabc6308dfdf4698a, State: Initialized, Role: FOLLOWER
I20260806 02:54:42.706740 17579 consensus_queue.cc:260] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.706641 17576 raft_consensus.cc:348] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.706982 17576 raft_consensus.cc:374] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:42.707042 17576 raft_consensus.cc:729] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0599c9a53c64494295c4d9601c1a7771, State: Initialized, Role: FOLLOWER
I20260806 02:54:42.707273 17577 raft_consensus.cc:348] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.707532 17577 raft_consensus.cc:374] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:42.707474 17576 consensus_queue.cc:260] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.707597 17577 raft_consensus.cc:729] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37c121e43364e6e83616aef54569ea5, State: Initialized, Role: FOLLOWER
I20260806 02:54:42.707934 17579 ts_tablet_manager.cc:1428] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:54:42.707984 17577 consensus_queue.cc:260] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.708336 17373 heartbeater.cc:503] Master 127.24.130.190:45541 was elected leader, sending a full tablet report...
I20260806 02:54:42.708505 17576 ts_tablet_manager.cc:1428] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:54:42.708846 17577 ts_tablet_manager.cc:1428] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:54:42.708865 17191 heartbeater.cc:503] Master 127.24.130.190:45541 was elected leader, sending a full tablet report...
I20260806 02:54:42.709170 17555 heartbeater.cc:503] Master 127.24.130.190:45541 was elected leader, sending a full tablet report...
W20260806 02:54:42.865420 17192 tablet.cc:2369] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:54:42.874797 17374 tablet.cc:2369] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:54:42.876075 17556 tablet.cc:2369] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:42.923986 17586 raft_consensus.cc:482] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:42.924278 17586 raft_consensus.cc:504] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.925595 17586 leader_election.cc:288] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe75c7cf1b054cceabc6308dfdf4698a (127.24.130.130:32991), 0599c9a53c64494295c4d9601c1a7771 (127.24.130.129:43505)
I20260806 02:54:42.930222 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.925878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:32991 (local address 127.24.130.131:57031)
0806 02:54:42.925999 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.926018 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.926130 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.926532 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.926535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.926555 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:42.926744 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.926754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.927701 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.927704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.928916 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.928928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.929803 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.929806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.929898 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.929922 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.929984 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.930043 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 02:54:42.930274 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.926015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:57031 (local address 127.24.130.130:32991)
0806 02:54:42.926172 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.926176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.926194 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:54:42.926259 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.926261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.926304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.926424 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:42.926868 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.927585 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.929069 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.929677 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.929811 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.929874 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.930037 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.930126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:54:42.930609 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.926467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:37299 (local address 127.24.130.129:43505)
0806 02:54:42.926604 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.926608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.926621 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:54:42.926795 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.926798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.926871 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.926960 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:42.927311 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.927939 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.929290 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.929999 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.930124 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.930169 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.930400 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.930503 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:54:42.930991 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.926178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43505 (local address 127.24.130.131:37299)
0806 02:54:42.926553 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:42.926568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:42.926682 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:42.927020 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:42.927023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:42.927055 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:42.927181 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.927186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.928075 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.928078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.929157 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:42.929165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.930135 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.930138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:42.930224 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.930260 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:54:42.930816 (+ 556us) client_negotiation.cc:240] Negotiation successful
0806 02:54:42.930885 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":164,"threads_started":1}
I20260806 02:54:42.931048 17307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9007b48532664fd5ae99abcea69858ef" candidate_uuid: "a37c121e43364e6e83616aef54569ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" is_pre_election: true
I20260806 02:54:42.931442 17307 raft_consensus.cc:2454] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a37c121e43364e6e83616aef54569ea5 in term 0.
I20260806 02:54:42.931773 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9007b48532664fd5ae99abcea69858ef" candidate_uuid: "a37c121e43364e6e83616aef54569ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0599c9a53c64494295c4d9601c1a7771" is_pre_election: true
I20260806 02:54:42.932034 17398 leader_election.cc:302] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a37c121e43364e6e83616aef54569ea5, fe75c7cf1b054cceabc6308dfdf4698a; no voters:
I20260806 02:54:42.932123 17121 raft_consensus.cc:2454] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a37c121e43364e6e83616aef54569ea5 in term 0.
I20260806 02:54:42.932328 17586 raft_consensus.cc:2781] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:42.932446 17586 raft_consensus.cc:482] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:42.932510 17586 raft_consensus.cc:3037] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:42.933791 17586 raft_consensus.cc:504] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.934185 17586 leader_election.cc:288] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [CANDIDATE]: Term 1 election: Requested vote from peers fe75c7cf1b054cceabc6308dfdf4698a (127.24.130.130:32991), 0599c9a53c64494295c4d9601c1a7771 (127.24.130.129:43505)
I20260806 02:54:42.934842 17307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9007b48532664fd5ae99abcea69858ef" candidate_uuid: "a37c121e43364e6e83616aef54569ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe75c7cf1b054cceabc6308dfdf4698a"
I20260806 02:54:42.934906 17121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9007b48532664fd5ae99abcea69858ef" candidate_uuid: "a37c121e43364e6e83616aef54569ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0599c9a53c64494295c4d9601c1a7771"
I20260806 02:54:42.935004 17307 raft_consensus.cc:3037] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:42.935060 17121 raft_consensus.cc:3037] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:42.936334 17307 raft_consensus.cc:2454] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a37c121e43364e6e83616aef54569ea5 in term 1.
I20260806 02:54:42.936368 17121 raft_consensus.cc:2454] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a37c121e43364e6e83616aef54569ea5 in term 1.
I20260806 02:54:42.936743 17398 leader_election.cc:302] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0599c9a53c64494295c4d9601c1a7771, a37c121e43364e6e83616aef54569ea5; no voters:
I20260806 02:54:42.936972 17586 raft_consensus.cc:2781] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:42.937297 17586 raft_consensus.cc:686] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 1 LEADER]: Becoming Leader. State: Replica: a37c121e43364e6e83616aef54569ea5, State: Running, Role: LEADER
I20260806 02:54:42.937739 17586 consensus_queue.cc:237] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } }
I20260806 02:54:42.941385 16911 catalog_manager.cc:5495] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a37c121e43364e6e83616aef54569ea5 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "a37c121e43364e6e83616aef54569ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a37c121e43364e6e83616aef54569ea5" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 } health_report { overall_health: UNKNOWN } } }
I20260806 02:54:42.979351 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:42.975473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.24.130.131:38559)
0806 02:54:42.975631 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:42.975636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:42.975653 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:54:42.975790 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:42.975793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:42.975858 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:42.975980 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:42.976268 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.976782 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.977667 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:42.977894 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:42.978004 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:42.978356 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:42.978890 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:42.979033 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:42.979098 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:54:42.979166 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:54:42.985775 17307 raft_consensus.cc:1261] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 1 FOLLOWER]: Refusing update from remote peer a37c121e43364e6e83616aef54569ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:42.985769 17121 raft_consensus.cc:1261] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 1 FOLLOWER]: Refusing update from remote peer a37c121e43364e6e83616aef54569ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:42.986626 17586 consensus_queue.cc:1035] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe75c7cf1b054cceabc6308dfdf4698a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:42.986991 17590 consensus_queue.cc:1035] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0599c9a53c64494295c4d9601c1a7771" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:42.994480 17594 mvcc.cc:205] Tried to move back new op lower bound from 7315394080701362176 to 7315394080516055040. Current Snapshot: MvccSnapshot[applied={T|T < 7315394080701362176}]
I20260806 02:54:42.997139 17597 mvcc.cc:205] Tried to move back new op lower bound from 7315394080701362176 to 7315394080516055040. Current Snapshot: MvccSnapshot[applied={T|T < 7315394080701362176}]
I20260806 02:54:43.002590 17596 mvcc.cc:205] Tried to move back new op lower bound from 7315394080701362176 to 7315394080516055040. Current Snapshot: MvccSnapshot[applied={T|T < 7315394080701362176}]
I20260806 02:54:43.003428 16911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:43.003626 16911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:43.006278 16911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a366996d33254a2da357e4afebe7ee2b: Sending DeleteTablet for 3 replicas of tablet 9007b48532664fd5ae99abcea69858ef
I20260806 02:54:43.007273 17282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9007b48532664fd5ae99abcea69858ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:43 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:44542
I20260806 02:54:43.007493 17465 tablet_service.cc:1514] Processing DeleteTablet for tablet 9007b48532664fd5ae99abcea69858ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:43 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:43068
I20260806 02:54:43.007534 17092 tablet_service.cc:1514] Processing DeleteTablet for tablet 9007b48532664fd5ae99abcea69858ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:43 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:50660
I20260806 02:54:43.007978 17605 tablet_replica.cc:330] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: stopping tablet replica
I20260806 02:54:43.008131 17608 tablet_replica.cc:330] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771: stopping tablet replica
I20260806 02:54:43.008354 17605 raft_consensus.cc:2229] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:43.008550 17608 raft_consensus.cc:2229] T 9007b48532664fd5ae99abcea69858ef P 0599c9a53c64494295c4d9601c1a7771 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:43.008599 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 17000
I20260806 02:54:43.008731 17605 raft_consensus.cc:2258] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:43.010475 17605 ts_tablet_manager.cc:1905] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:43.013540 17605 ts_tablet_manager.cc:1918] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:43.013655 17605 log.cc:1196] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/9007b48532664fd5ae99abcea69858ef
I20260806 02:54:43.014003 17605 ts_tablet_manager.cc:1939] T 9007b48532664fd5ae99abcea69858ef P fe75c7cf1b054cceabc6308dfdf4698a: Deleting consensus metadata
I20260806 02:54:43.015491 16889 catalog_manager.cc:4841] TS fe75c7cf1b054cceabc6308dfdf4698a (127.24.130.130:32991): tablet 9007b48532664fd5ae99abcea69858ef (table test-table [id=916eb7b12acb417dad9e17d182d1326d]) successfully deleted
I20260806 02:54:43.019011 17606 tablet_replica.cc:330] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: stopping tablet replica
I20260806 02:54:43.019419 17606 raft_consensus.cc:2229] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:43.019851 17606 raft_consensus.cc:2258] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:43.021492 17606 ts_tablet_manager.cc:1905] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:43.024226 17606 ts_tablet_manager.cc:1918] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:43.024377 17606 log.cc:1196] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/9007b48532664fd5ae99abcea69858ef
W20260806 02:54:43.024989 16889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:43505 (error 108)
I20260806 02:54:43.025223 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 17197
I20260806 02:54:43.031937 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.031502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:43505 (local address 127.0.0.1:50676)
0806 02:54:43.031701 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:43.031805 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 02:54:43.032174 16889 catalog_manager.cc:4584] TS 0599c9a53c64494295c4d9601c1a7771 (127.24.130.129:43505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9007b48532664fd5ae99abcea69858ef: Network error: Client connection negotiation failed: client connection to 127.24.130.129:43505: connect: Connection refused (error 111)
I20260806 02:54:43.036237 17606 ts_tablet_manager.cc:1939] T 9007b48532664fd5ae99abcea69858ef P a37c121e43364e6e83616aef54569ea5: Deleting consensus metadata
I20260806 02:54:43.038404 16889 catalog_manager.cc:4841] TS a37c121e43364e6e83616aef54569ea5 (127.24.130.131:38559): tablet 9007b48532664fd5ae99abcea69858ef (table test-table [id=916eb7b12acb417dad9e17d182d1326d]) successfully deleted
I20260806 02:54:43.045687 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 17379
I20260806 02:54:43.058833 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 16866
2026-08-06T02:54:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2838 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T02:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:43Z Disabled control of system clock
I20260806 02:54:43.099254 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:46851
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:54:43.219839 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:43.220170 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:43.220208 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:43.224445 17616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:43.224540 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:43.224556 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:43.224574 17616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:43.224591 17616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:43.229904 17616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:46851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:43.231210 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:43.232731 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:43.242324 17624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.242347 17626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.242357 17623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:43.242511 17616 server_base.cc:1034] running on GCE node
I20260806 02:54:43.243331 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:43.244544 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:43.245734 17616 hybrid_clock.cc:648] HybridClock initialized: now 1785984883245706 us; error 53 us; skew 500 ppm
I20260806 02:54:43.247895 17616 webserver.cc:460] Webserver started at http://127.24.130.190:39211/ using document root <none> and password file <none>
I20260806 02:54:43.248479 17616 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:43.248544 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:43.248725 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:43.251006 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "72847aec47394352b5391d1d87fc9c47"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "ccd832540bad37ecf855860e9b9d82e3"
server_key_iv: "177f3424c9986f00da2d9c14e7045787"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.251582 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "72847aec47394352b5391d1d87fc9c47"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "ccd832540bad37ecf855860e9b9d82e3"
server_key_iv: "177f3424c9986f00da2d9c14e7045787"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.255645 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:54:43.258226 17635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.259604 17616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:54:43.259763 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "72847aec47394352b5391d1d87fc9c47"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "ccd832540bad37ecf855860e9b9d82e3"
server_key_iv: "177f3424c9986f00da2d9c14e7045787"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.259899 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:43.285630 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:43.286491 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:43.286775 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:43.294395 17705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:46851 every 8 connection(s)
I20260806 02:54:43.294385 17616 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:46851
I20260806 02:54:43.295643 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:43.297452 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 17616
I20260806 02:54:43.297603 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:43.297935 25098 external_mini_cluster.cc:1428] Setting key e6f2187e21871dc6d27fac24b1b7a8c9
I20260806 02:54:43.299734 17706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:43.306879 17706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47: Bootstrap starting.
I20260806 02:54:43.308719 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.299590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.24.130.190:46851)
0806 02:54:43.300351 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:43.300373 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:43.300414 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:54:43.301131 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:43.301149 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:43.301564 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:43.301881 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:43.302254 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.303520 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.304861 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.305377 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.305512 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.306600 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:43.306606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:43.306619 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:43.307231 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:43.307300 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:43.307323 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:43.307408 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:43.307620 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:54:43.307965 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":200,"threads_started":1}
I20260806 02:54:43.309448 17706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:43.310268 17706 log.cc:824] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:43.312532 17706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47: No bootstrap required, opened a new log
I20260806 02:54:43.316417 17706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72847aec47394352b5391d1d87fc9c47" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46851 } }
I20260806 02:54:43.316704 17706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:43.316792 17706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72847aec47394352b5391d1d87fc9c47, State: Initialized, Role: FOLLOWER
I20260806 02:54:43.317245 17706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72847aec47394352b5391d1d87fc9c47" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46851 } }
I20260806 02:54:43.317425 17706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:43.317512 17706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:43.317642 17706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:43.318790 17706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72847aec47394352b5391d1d87fc9c47" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46851 } }
I20260806 02:54:43.319171 17706 leader_election.cc:302] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72847aec47394352b5391d1d87fc9c47; no voters:
I20260806 02:54:43.319547 17706 leader_election.cc:288] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:43.319700 17714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:43.319993 17714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [term 1 LEADER]: Becoming Leader. State: Replica: 72847aec47394352b5391d1d87fc9c47, State: Running, Role: LEADER
I20260806 02:54:43.320341 17714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72847aec47394352b5391d1d87fc9c47" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46851 } }
I20260806 02:54:43.320958 17706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:43.322110 17717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72847aec47394352b5391d1d87fc9c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72847aec47394352b5391d1d87fc9c47" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46851 } } }
I20260806 02:54:43.322256 17717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:43.322543 17718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72847aec47394352b5391d1d87fc9c47. Latest consensus state: current_term: 1 leader_uuid: "72847aec47394352b5391d1d87fc9c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72847aec47394352b5391d1d87fc9c47" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 46851 } } }
I20260806 02:54:43.322640 17718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:43.322938 17730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:43.326022 17730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:43.332336 17730 catalog_manager.cc:1347] Generated new cluster ID: 3ddfbeed3dc74bdc96ea8d6aac83b7bb
I20260806 02:54:43.332435 17730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:43.347240 17730 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:43.348300 17730 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:43.356910 17730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47: Generated new TSK 0
I20260806 02:54:43.357762 17730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:43.362150 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46851
--builtin_ntp_servers=127.24.130.148:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:54:43.484934 17745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:43.485267 17745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:43.485309 17745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:43.489563 17745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:43.489652 17745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:43.489711 17745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:43.494859 17745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:43.496147 17745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:43.497666 17745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:43.508010 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.508010 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.508060 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:43.508316 17745 server_base.cc:1034] running on GCE node
I20260806 02:54:43.508914 17745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:43.509701 17745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:43.510895 17745 hybrid_clock.cc:648] HybridClock initialized: now 1785984883510851 us; error 56 us; skew 500 ppm
I20260806 02:54:43.513024 17745 webserver.cc:460] Webserver started at http://127.24.130.129:46247/ using document root <none> and password file <none>
I20260806 02:54:43.513674 17745 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:43.513744 17745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:43.514034 17745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:43.516377 17745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "badc6718af7645bdb7df6ffb0ddd7ba8"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "7f43f36f0cdc835dab999354f94bae68"
server_key_iv: "4396c6b91896910fd9a59b3021c0a336"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.517043 17745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "badc6718af7645bdb7df6ffb0ddd7ba8"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "7f43f36f0cdc835dab999354f94bae68"
server_key_iv: "4396c6b91896910fd9a59b3021c0a336"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.521216 17745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:54:43.524037 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.525357 17745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:54:43.525498 17745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "badc6718af7645bdb7df6ffb0ddd7ba8"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "7f43f36f0cdc835dab999354f94bae68"
server_key_iv: "4396c6b91896910fd9a59b3021c0a336"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.525628 17745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:43.543357 17745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:43.544176 17745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:43.544420 17745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:43.545121 17745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:43.546205 17745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:43.546259 17745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.546336 17745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:43.546368 17745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.556154 17745 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:36785
I20260806 02:54:43.556175 17925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:36785 every 8 connection(s)
I20260806 02:54:43.557192 17745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:43.561424 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 17745
I20260806 02:54:43.561549 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:43.561877 25098 external_mini_cluster.cc:1428] Setting key 5569d94526f6a97781b3b97ed3618442
I20260806 02:54:43.564088 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46851
--builtin_ntp_servers=127.24.130.148:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:54:43.568187 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.559289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:53023 (local address 127.24.130.190:46851)
0806 02:54:43.559461 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:43.559466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:43.560634 (+ 1168us) server_negotiation.cc:407] Connection header received
0806 02:54:43.562270 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:43.562274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:43.562346 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:43.562471 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:43.563653 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.564290 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.565580 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.565960 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.566086 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.567519 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:43.567526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:43.567532 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:43.567552 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:43.567579 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:43.567592 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:43.567679 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:43.567866 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 02:54:43.567965 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:54:43.569005 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.559747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:46851 (local address 127.24.130.129:53023)
0806 02:54:43.560460 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:43.560506 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:43.562016 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:43.562742 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:43.562751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:43.562850 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:43.563506 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:43.563524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.564449 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.564453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:43.565441 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:43.565452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.566139 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.566143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:43.566349 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.567159 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:43.567162 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:43.567226 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:43.567316 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:43.567697 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:43.567707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:43.567715 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:54:43.567830 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:54:43.568081 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":146,"threads_started":1}
I20260806 02:54:43.570461 17926 heartbeater.cc:347] Connected to a master server at 127.24.130.190:46851
I20260806 02:54:43.570797 17926 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:43.571575 17926 heartbeater.cc:511] Master 127.24.130.190:46851 requested a full tablet report, sending...
I20260806 02:54:43.573423 17661 ts_manager.cc:194] Registered new tserver with Master: badc6718af7645bdb7df6ffb0ddd7ba8 (127.24.130.129:36785)
I20260806 02:54:43.574633 17661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.129:53023
W20260806 02:54:43.685457 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:43.685781 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:43.685828 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:43.690143 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:43.690234 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:43.690294 17929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:43.695462 17929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:43.696714 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:43.698201 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:43.709961 17937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.710033 17940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.710078 17938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:43.710414 17929 server_base.cc:1034] running on GCE node
I20260806 02:54:43.710906 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:43.711591 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:43.712766 17929 hybrid_clock.cc:648] HybridClock initialized: now 1785984883712736 us; error 51 us; skew 500 ppm
I20260806 02:54:43.715085 17929 webserver.cc:460] Webserver started at http://127.24.130.130:44009/ using document root <none> and password file <none>
I20260806 02:54:43.715729 17929 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:43.715803 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:43.716066 17929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:43.718321 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "4757365ba9a6473d9c769a4aaa965adc"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "7b83fac5ecd345793381c96a300cf5df"
server_key_iv: "75e4329bdbd0752c6b7458568df41cc8"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.719010 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "4757365ba9a6473d9c769a4aaa965adc"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "7b83fac5ecd345793381c96a300cf5df"
server_key_iv: "75e4329bdbd0752c6b7458568df41cc8"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.723377 17929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:54:43.726007 17948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.727382 17929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:54:43.727550 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "4757365ba9a6473d9c769a4aaa965adc"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "7b83fac5ecd345793381c96a300cf5df"
server_key_iv: "75e4329bdbd0752c6b7458568df41cc8"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.727690 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:43.744118 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:43.744935 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:43.745175 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:43.745857 17929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:43.746991 17929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:43.747067 17929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.747148 17929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:43.747190 17929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.756824 17929 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:38355
I20260806 02:54:43.756861 18107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:38355 every 8 connection(s)
I20260806 02:54:43.757858 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:43.763247 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 17929
I20260806 02:54:43.763399 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:43.763737 25098 external_mini_cluster.cc:1428] Setting key 51a9d0efc6f96f5319abe3401a26dff5
I20260806 02:54:43.766153 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46851
--builtin_ntp_servers=127.24.130.148:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:54:43.769156 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.759932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:42295 (local address 127.24.130.190:46851)
0806 02:54:43.760126 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:43.760130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:43.761292 (+ 1162us) server_negotiation.cc:407] Connection header received
0806 02:54:43.762807 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:43.762810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:43.762862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:43.763016 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:43.764381 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.764998 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.766766 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.766992 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.767084 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.768542 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:43.768549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:43.768555 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:43.768574 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:43.768603 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:43.768615 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:43.768708 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:43.768865 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:54:43.768953 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:54:43.770031 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.760375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:46851 (local address 127.24.130.130:42295)
0806 02:54:43.761108 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:43.761155 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:43.762508 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:43.763252 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:43.763262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:43.763401 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:43.764176 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:43.764203 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.765162 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.765164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:43.766569 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:43.766585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.767168 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.767175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:43.767435 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.768229 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:43.768231 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:43.768275 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:43.768351 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:43.768715 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:43.768722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:43.768727 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:54:43.768831 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:54:43.769090 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
I20260806 02:54:43.771546 18108 heartbeater.cc:347] Connected to a master server at 127.24.130.190:46851
I20260806 02:54:43.771849 18108 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:43.772446 18108 heartbeater.cc:511] Master 127.24.130.190:46851 requested a full tablet report, sending...
I20260806 02:54:43.773667 17661 ts_manager.cc:194] Registered new tserver with Master: 4757365ba9a6473d9c769a4aaa965adc (127.24.130.130:38355)
I20260806 02:54:43.774273 17661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.130:42295
W20260806 02:54:43.890360 18111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:43.890791 18111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:43.890883 18111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:43.895184 18111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:43.895282 18111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:43.895385 18111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:43.900399 18111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:46851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:43.901683 18111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:43.903375 18111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:54:43.914193 18111 server_base.cc:1034] running on GCE node
W20260806 02:54:43.914172 18121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.914417 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:43.914149 18119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:43.914981 18111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:43.915760 18111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:43.916949 18111 hybrid_clock.cc:648] HybridClock initialized: now 1785984883916932 us; error 35 us; skew 500 ppm
I20260806 02:54:43.919085 18111 webserver.cc:460] Webserver started at http://127.24.130.131:41225/ using document root <none> and password file <none>
I20260806 02:54:43.919723 18111 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:43.919823 18111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:43.920078 18111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:43.922514 18111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "35b9c3138b3d4193bf47321660d3cca4"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "d29997ef1ae6775944e89d88429f1cce"
server_key_iv: "5928513261121926e1445c32e1a736c9"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.923204 18111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "35b9c3138b3d4193bf47321660d3cca4"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "d29997ef1ae6775944e89d88429f1cce"
server_key_iv: "5928513261121926e1445c32e1a736c9"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.927186 18111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:54:43.929903 18133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.931351 18111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:43.931529 18111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "35b9c3138b3d4193bf47321660d3cca4"
format_stamp: "Formatted at 2026-08-06 02:54:43 on dist-test-slave-3kk6"
server_key: "d29997ef1ae6775944e89d88429f1cce"
server_key_iv: "5928513261121926e1445c32e1a736c9"
server_key_version: "encryptionkey@0"
I20260806 02:54:43.931679 18111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:43.948770 18111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:43.949625 18111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:43.949872 18111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:43.950580 18111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:43.951711 18111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:43.951798 18111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.951879 18111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:43.951912 18111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:43.961820 18111 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:38607
I20260806 02:54:43.961841 18293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:38607 every 8 connection(s)
I20260806 02:54:43.962973 18111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:43.965534 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 18111
I20260806 02:54:43.965667 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:43.965967 25098 external_mini_cluster.cc:1428] Setting key f8b3bdc530cc5d736ec2b7a268b536e4
I20260806 02:54:43.973678 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.965345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:44731 (local address 127.24.130.190:46851)
0806 02:54:43.965524 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:43.965529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:43.966915 (+ 1386us) server_negotiation.cc:407] Connection header received
0806 02:54:43.968360 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:43.968363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:43.968421 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:43.968492 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:43.969475 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.970009 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.971405 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.971711 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.971835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.973127 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:43.973131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:43.973134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:43.973148 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:43.973180 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:43.973188 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:43.973271 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:43.973445 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:54:43.973520 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:54:43.974618 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.965862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:46851 (local address 127.24.130.131:44731)
0806 02:54:43.966746 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:43.966809 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:43.968167 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:43.968670 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:43.968677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:43.968804 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:43.969339 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:43.969353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.970157 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.970160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:43.971253 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:43.971265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.971890 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.971894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:43.972080 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.972818 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:43.972820 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:43.972869 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:43.972948 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:43.973275 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:43.973287 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:43.973294 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:54:43.973445 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 02:54:43.973771 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":205,"threads_started":1}
I20260806 02:54:43.975924 18295 heartbeater.cc:347] Connected to a master server at 127.24.130.190:46851
I20260806 02:54:43.976233 18295 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:43.976848 18295 heartbeater.cc:511] Master 127.24.130.190:46851 requested a full tablet report, sending...
I20260806 02:54:43.978026 17661 ts_manager.cc:194] Registered new tserver with Master: 35b9c3138b3d4193bf47321660d3cca4 (127.24.130.131:38607)
I20260806 02:54:43.978775 17661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.131:44731
I20260806 02:54:43.980355 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:43.986503 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:43.982542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.24.130.190:46851)
0806 02:54:43.982766 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:43.982771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:43.982960 (+ 189us) server_negotiation.cc:407] Connection header received
0806 02:54:43.983069 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:43.983071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:43.983113 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:43.983184 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:43.983485 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.984018 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.984832 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:43.985055 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:43.985143 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:43.986027 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:43.986030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:43.986032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:43.986045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:43.986065 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:43.986073 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:43.986144 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:43.986238 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 02:54:43.986302 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:54:43.989873 17661 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:43.993121 17661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:44.008701 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.002454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36785 (local address 127.0.0.1:53662)
0806 02:54:44.003017 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:44.003072 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:44.003230 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:44.004188 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:44.004195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:44.004222 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:44.004362 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.004369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.005595 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.005599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.007037 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.007046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.008133 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.008137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.008243 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.008280 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:54:44.008411 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:54:44.008540 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":80,"thread_start_us":89,"threads_started":1}
I20260806 02:54:44.008706 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.002618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.24.130.129:36785)
0806 02:54:44.003552 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:44.003565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:44.003591 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:54:44.003677 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:44.003682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:44.003901 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:44.004062 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:44.004500 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.005472 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.007184 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.008016 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.008148 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.008347 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:44.008428 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 02:54:44.008529 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":172,"threads_started":1}
I20260806 02:54:44.008980 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.002683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:38355 (local address 127.0.0.1:56846)
0806 02:54:44.003088 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:44.003107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:44.003218 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:44.004585 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:44.004588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:44.004607 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:44.004703 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.004710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.006347 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.006350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.007448 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.007458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.008557 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.008561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.008651 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.008675 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:54:44.008766 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:54:44.008837 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":104,"threads_started":1}
I20260806 02:54:44.009496 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.003315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.24.130.130:38355)
0806 02:54:44.003756 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:44.003766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:44.003785 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:54:44.003849 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:44.003855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:44.004407 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:44.004592 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:44.004979 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.006173 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.007574 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.008202 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.008327 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.008488 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:44.009220 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 02:54:44.009325 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":141,"threads_started":1}
I20260806 02:54:44.009842 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.003239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:38607 (local address 127.0.0.1:53456)
0806 02:54:44.003948 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:44.003966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:44.004056 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:44.004749 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:44.004752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:44.004770 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:44.004869 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.004875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.006309 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.006312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.007878 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.007889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.009437 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.009440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.009534 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.009564 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:54:44.009643 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:54:44.009718 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":346,"thread_start_us":113,"threads_started":1}
I20260806 02:54:44.010408 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.003562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.24.130.131:38607)
0806 02:54:44.004146 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:44.004160 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:44.004182 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:54:44.004264 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:44.004268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:44.004455 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:44.004647 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:44.004997 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.006173 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.008034 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.008814 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.009188 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.009366 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:44.010187 (+ 821us) server_negotiation.cc:299] Negotiation successful
0806 02:54:44.010286 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":105,"threads_started":1}
I20260806 02:54:44.012434 18015 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd0adb28e97406d9598152f4cf38d1c (DEFAULT_TABLE table=test-table [id=1d840f05324549bfaca8e11f40c21db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:44.012324 17830 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd0adb28e97406d9598152f4cf38d1c (DEFAULT_TABLE table=test-table [id=1d840f05324549bfaca8e11f40c21db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:44.013000 18199 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd0adb28e97406d9598152f4cf38d1c (DEFAULT_TABLE table=test-table [id=1d840f05324549bfaca8e11f40c21db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:44.013456 18015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd0adb28e97406d9598152f4cf38d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:44.013471 17830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd0adb28e97406d9598152f4cf38d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:44.013963 18199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd0adb28e97406d9598152f4cf38d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:44.019963 18317 tablet_bootstrap.cc:492] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Bootstrap starting.
I20260806 02:54:44.021508 18318 tablet_bootstrap.cc:492] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: Bootstrap starting.
I20260806 02:54:44.021871 18319 tablet_bootstrap.cc:492] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Bootstrap starting.
I20260806 02:54:44.022467 18317 tablet_bootstrap.cc:654] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:44.023502 18317 log.cc:824] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:44.023775 18318 tablet_bootstrap.cc:654] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:44.024230 18319 tablet_bootstrap.cc:654] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:44.024531 18318 log.cc:824] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:44.024823 18319 log.cc:824] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:44.026611 18317 tablet_bootstrap.cc:492] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: No bootstrap required, opened a new log
I20260806 02:54:44.026729 18319 tablet_bootstrap.cc:492] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: No bootstrap required, opened a new log
I20260806 02:54:44.026749 18318 tablet_bootstrap.cc:492] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: No bootstrap required, opened a new log
I20260806 02:54:44.026952 18318 ts_tablet_manager.cc:1397] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:54:44.026952 18319 ts_tablet_manager.cc:1397] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:54:44.027097 18317 ts_tablet_manager.cc:1397] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 02:54:44.030633 18317 raft_consensus.cc:348] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.030944 18317 raft_consensus.cc:374] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:44.030983 18317 raft_consensus.cc:729] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4757365ba9a6473d9c769a4aaa965adc, State: Initialized, Role: FOLLOWER
I20260806 02:54:44.031453 18317 consensus_queue.cc:260] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.031313 18318 raft_consensus.cc:348] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.031414 18319 raft_consensus.cc:348] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.031595 18318 raft_consensus.cc:374] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:44.031615 18319 raft_consensus.cc:374] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:44.031652 18318 raft_consensus.cc:729] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badc6718af7645bdb7df6ffb0ddd7ba8, State: Initialized, Role: FOLLOWER
I20260806 02:54:44.031750 18319 raft_consensus.cc:729] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b9c3138b3d4193bf47321660d3cca4, State: Initialized, Role: FOLLOWER
I20260806 02:54:44.032214 18319 consensus_queue.cc:260] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.032572 18317 ts_tablet_manager.cc:1428] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260806 02:54:44.032795 18318 consensus_queue.cc:260] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.033314 18319 ts_tablet_manager.cc:1428] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 02:54:44.033314 18108 heartbeater.cc:503] Master 127.24.130.190:46851 was elected leader, sending a full tablet report...
I20260806 02:54:44.033747 18295 heartbeater.cc:503] Master 127.24.130.190:46851 was elected leader, sending a full tablet report...
I20260806 02:54:44.033865 18318 ts_tablet_manager.cc:1428] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260806 02:54:44.034394 17926 heartbeater.cc:503] Master 127.24.130.190:46851 was elected leader, sending a full tablet report...
W20260806 02:54:44.058501 17927 tablet.cc:2369] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:44.108814 18325 raft_consensus.cc:482] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:44.109103 18325 raft_consensus.cc:504] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.110450 18325 leader_election.cc:288] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers badc6718af7645bdb7df6ffb0ddd7ba8 (127.24.130.129:36785), 4757365ba9a6473d9c769a4aaa965adc (127.24.130.130:38355)
I20260806 02:54:44.114519 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.110857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:48851 (local address 127.24.130.130:38355)
0806 02:54:44.111002 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:44.111006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:44.111290 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:54:44.111459 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:44.111462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:44.111514 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:44.111594 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:44.111918 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.112409 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.113529 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.113931 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.114069 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.114124 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:44.114312 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 02:54:44.114385 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:54:44.114537 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.110762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:38355 (local address 127.24.130.131:48851)
0806 02:54:44.111186 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:44.111198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:44.111332 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:44.111619 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:44.111621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:44.111637 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:44.111788 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.111793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.112529 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.112534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.113389 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.113406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.114067 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.114070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.114169 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.114201 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:54:44.114293 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:54:44.114367 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":25,"thread_start_us":168,"threads_started":1}
I20260806 02:54:44.114694 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.110721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36785 (local address 127.24.130.131:45945)
0806 02:54:44.110934 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:44.110953 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:44.111070 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:44.111389 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:44.111393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:44.111411 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:44.111549 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.111556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.112384 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.112390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.113388 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:44.113399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.114172 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.114176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:44.114300 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.114319 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:54:44.114411 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:54:44.114493 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":64}
I20260806 02:54:44.114694 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.110844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:45945 (local address 127.24.130.129:36785)
0806 02:54:44.110982 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:44.110989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:44.111013 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:54:44.111218 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:44.111221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:44.111273 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:44.111364 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:44.111688 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.112255 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.113529 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.114023 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.114171 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.114226 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:44.114448 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:54:44.114529 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:54:44.115402 18045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd0adb28e97406d9598152f4cf38d1c" candidate_uuid: "35b9c3138b3d4193bf47321660d3cca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4757365ba9a6473d9c769a4aaa965adc" is_pre_election: true
I20260806 02:54:44.115402 17856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd0adb28e97406d9598152f4cf38d1c" candidate_uuid: "35b9c3138b3d4193bf47321660d3cca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" is_pre_election: true
I20260806 02:54:44.115770 18045 raft_consensus.cc:2454] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35b9c3138b3d4193bf47321660d3cca4 in term 0.
I20260806 02:54:44.115772 17856 raft_consensus.cc:2454] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35b9c3138b3d4193bf47321660d3cca4 in term 0.
I20260806 02:54:44.116287 18135 leader_election.cc:302] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b9c3138b3d4193bf47321660d3cca4, 4757365ba9a6473d9c769a4aaa965adc; no voters:
I20260806 02:54:44.116618 18325 raft_consensus.cc:2781] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:44.116727 18325 raft_consensus.cc:482] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:44.116822 18325 raft_consensus.cc:3037] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:44.117990 18325 raft_consensus.cc:504] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.118386 18325 leader_election.cc:288] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [CANDIDATE]: Term 1 election: Requested vote from peers badc6718af7645bdb7df6ffb0ddd7ba8 (127.24.130.129:36785), 4757365ba9a6473d9c769a4aaa965adc (127.24.130.130:38355)
I20260806 02:54:44.118798 17856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd0adb28e97406d9598152f4cf38d1c" candidate_uuid: "35b9c3138b3d4193bf47321660d3cca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8"
I20260806 02:54:44.118870 18045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd0adb28e97406d9598152f4cf38d1c" candidate_uuid: "35b9c3138b3d4193bf47321660d3cca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4757365ba9a6473d9c769a4aaa965adc"
I20260806 02:54:44.118955 17856 raft_consensus.cc:3037] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:44.119037 18045 raft_consensus.cc:3037] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:44.120193 18045 raft_consensus.cc:2454] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35b9c3138b3d4193bf47321660d3cca4 in term 1.
I20260806 02:54:44.120194 17856 raft_consensus.cc:2454] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35b9c3138b3d4193bf47321660d3cca4 in term 1.
I20260806 02:54:44.120533 18135 leader_election.cc:302] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b9c3138b3d4193bf47321660d3cca4, 4757365ba9a6473d9c769a4aaa965adc; no voters:
I20260806 02:54:44.120738 18325 raft_consensus.cc:2781] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:44.121009 18325 raft_consensus.cc:686] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 1 LEADER]: Becoming Leader. State: Replica: 35b9c3138b3d4193bf47321660d3cca4, State: Running, Role: LEADER
I20260806 02:54:44.121462 18325 consensus_queue.cc:237] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } }
I20260806 02:54:44.125219 17659 catalog_manager.cc:5495] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35b9c3138b3d4193bf47321660d3cca4 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "35b9c3138b3d4193bf47321660d3cca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35b9c3138b3d4193bf47321660d3cca4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 38607 } health_report { overall_health: HEALTHY } } }
I20260806 02:54:44.164912 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.161321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.24.130.131:38607)
0806 02:54:44.161458 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:44.161462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:44.161475 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:54:44.161556 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:44.161558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:44.161613 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:44.161712 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:44.161941 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.162453 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.163390 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:44.163606 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:44.163687 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:44.163975 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:44.164465 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:44.164557 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:44.164699 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 02:54:44.164763 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:54:44.171196 18045 raft_consensus.cc:1261] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 1 FOLLOWER]: Refusing update from remote peer 35b9c3138b3d4193bf47321660d3cca4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:44.171196 17856 raft_consensus.cc:1261] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 1 FOLLOWER]: Refusing update from remote peer 35b9c3138b3d4193bf47321660d3cca4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:44.172045 18325 consensus_queue.cc:1035] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4757365ba9a6473d9c769a4aaa965adc" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:44.172297 18328 consensus_queue.cc:1035] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "badc6718af7645bdb7df6ffb0ddd7ba8" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:44.180982 18331 mvcc.cc:205] Tried to move back new op lower bound from 7315394085557166080 to 7315394085365125120. Current Snapshot: MvccSnapshot[applied={T|T < 7315394085557166080}]
I20260806 02:54:44.181260 18332 mvcc.cc:205] Tried to move back new op lower bound from 7315394085557166080 to 7315394085365125120. Current Snapshot: MvccSnapshot[applied={T|T < 7315394085557166080}]
I20260806 02:54:44.181641 18334 mvcc.cc:205] Tried to move back new op lower bound from 7315394085557166080 to 7315394085365125120. Current Snapshot: MvccSnapshot[applied={T|T < 7315394085557166080}]
I20260806 02:54:44.191236 17659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:44.191419 17659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:44.194109 17659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72847aec47394352b5391d1d87fc9c47: Sending DeleteTablet for 3 replicas of tablet 9fd0adb28e97406d9598152f4cf38d1c
I20260806 02:54:44.195103 17830 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fd0adb28e97406d9598152f4cf38d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:44 UTC) from {username='slave'} at 127.0.0.1:53662
I20260806 02:54:44.195099 18199 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fd0adb28e97406d9598152f4cf38d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:44 UTC) from {username='slave'} at 127.0.0.1:53456
I20260806 02:54:44.195223 18015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fd0adb28e97406d9598152f4cf38d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:44 UTC) from {username='slave'} at 127.0.0.1:56846
I20260806 02:54:44.195845 18345 tablet_replica.cc:330] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: stopping tablet replica
I20260806 02:54:44.196100 18346 tablet_replica.cc:330] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: stopping tablet replica
I20260806 02:54:44.196208 18343 tablet_replica.cc:330] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8: stopping tablet replica
I20260806 02:54:44.196506 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 17745
I20260806 02:54:44.196532 18346 raft_consensus.cc:2229] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:44.196636 18343 raft_consensus.cc:2229] T 9fd0adb28e97406d9598152f4cf38d1c P badc6718af7645bdb7df6ffb0ddd7ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:44.196913 18346 raft_consensus.cc:2258] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:44.197291 18345 raft_consensus.cc:2229] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:44.197749 18345 raft_consensus.cc:2258] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:44.203249 18345 ts_tablet_manager.cc:1905] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:44.206799 18345 ts_tablet_manager.cc:1918] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:44.206931 18345 log.cc:1196] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/9fd0adb28e97406d9598152f4cf38d1c
I20260806 02:54:44.207306 18345 ts_tablet_manager.cc:1939] T 9fd0adb28e97406d9598152f4cf38d1c P 35b9c3138b3d4193bf47321660d3cca4: Deleting consensus metadata
I20260806 02:54:44.211755 17639 catalog_manager.cc:4841] TS 35b9c3138b3d4193bf47321660d3cca4 (127.24.130.131:38607): tablet 9fd0adb28e97406d9598152f4cf38d1c (table test-table [id=1d840f05324549bfaca8e11f40c21db6]) successfully deleted
I20260806 02:54:44.213418 18346 ts_tablet_manager.cc:1905] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:44.216048 18346 ts_tablet_manager.cc:1918] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:44.216193 18346 log.cc:1196] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/9fd0adb28e97406d9598152f4cf38d1c
I20260806 02:54:44.216605 18346 ts_tablet_manager.cc:1939] T 9fd0adb28e97406d9598152f4cf38d1c P 4757365ba9a6473d9c769a4aaa965adc: Deleting consensus metadata
I20260806 02:54:44.218030 17636 catalog_manager.cc:4841] TS 4757365ba9a6473d9c769a4aaa965adc (127.24.130.130:38355): tablet 9fd0adb28e97406d9598152f4cf38d1c (table test-table [id=1d840f05324549bfaca8e11f40c21db6]) successfully deleted
W20260806 02:54:44.219323 17639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:36785 (error 108)
I20260806 02:54:44.219642 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 17929
I20260806 02:54:44.223201 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:44.222420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:36785 (local address 127.0.0.1:53668)
0806 02:54:44.222617 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:44.223063 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:36785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:54:44.223929 17639 catalog_manager.cc:4584] TS badc6718af7645bdb7df6ffb0ddd7ba8 (127.24.130.129:36785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fd0adb28e97406d9598152f4cf38d1c: Network error: Client connection negotiation failed: client connection to 127.24.130.129:36785: connect: Connection refused (error 111)
I20260806 02:54:44.230388 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 18111
I20260806 02:54:44.246920 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 17616
2026-08-06T02:54:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1187 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:44 dist-test-slave-3kk6 krb5kdc[18351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:44 dist-test-slave-3kk6 krb5kdc[18351](info): set up 2 sockets
Aug 06 02:54:44 dist-test-slave-3kk6 krb5kdc[18351](info): commencing operation
krb5kdc: starting...
W20260806 02:54:45.474953 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:45 dist-test-slave-3kk6 krb5kdc[18351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:54:45.616127 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43409
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:45.740442 18370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:45.740845 18370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:45.740931 18370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:45.745273 18370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:45.745404 18370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:45.745442 18370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:45.745482 18370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:45.745543 18370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:45.751332 18370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:43409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:45.752650 18370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:45.753835 18370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:54:45.763494 18370 server_base.cc:1034] running on GCE node
W20260806 02:54:45.763511 18381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:45.763494 18378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:45.763491 18377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:45.764487 18370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:45.765816 18370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:45.767024 18370 hybrid_clock.cc:648] HybridClock initialized: now 1785984885766989 us; error 59 us; skew 500 ppm
Aug 06 02:54:45 dist-test-slave-3kk6 krb5kdc[18351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984885, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:45.774796 18370 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:54:45.775885 18370 webserver.cc:460] Webserver started at http://127.24.130.190:41223/ using document root <none> and password file <none>
I20260806 02:54:45.776477 18370 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:45.776546 18370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:45.776727 18370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:45.779091 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "ae2c879673a742ce8dd7cf35b3d09041"
format_stamp: "Formatted at 2026-08-06 02:54:45 on dist-test-slave-3kk6"
server_key: "6e58f5cfbd7a7af02e3cb1fe2dda1a79"
server_key_iv: "79bd955afaabc6aff9fd9e6588c8a26b"
server_key_version: "encryptionkey@0"
I20260806 02:54:45.779688 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "ae2c879673a742ce8dd7cf35b3d09041"
format_stamp: "Formatted at 2026-08-06 02:54:45 on dist-test-slave-3kk6"
server_key: "6e58f5cfbd7a7af02e3cb1fe2dda1a79"
server_key_iv: "79bd955afaabc6aff9fd9e6588c8a26b"
server_key_version: "encryptionkey@0"
I20260806 02:54:45.783793 18370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:54:45.786509 18392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:45.787881 18370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:54:45.788043 18370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "ae2c879673a742ce8dd7cf35b3d09041"
format_stamp: "Formatted at 2026-08-06 02:54:45 on dist-test-slave-3kk6"
server_key: "6e58f5cfbd7a7af02e3cb1fe2dda1a79"
server_key_iv: "79bd955afaabc6aff9fd9e6588c8a26b"
server_key_version: "encryptionkey@0"
I20260806 02:54:45.788175 18370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:45.803812 18370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:45.807677 18370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:45.807917 18370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:45.816169 18370 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:43409
I20260806 02:54:45.816190 18474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:43409 every 8 connection(s)
I20260806 02:54:45.817256 18370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:45.820493 18476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:45.825275 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 18370
I20260806 02:54:45.825418 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:45.825770 25098 external_mini_cluster.cc:1428] Setting key 4472dfe5975050da04169bd407f03053
I20260806 02:54:45.827271 18476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041: Bootstrap starting.
I20260806 02:54:45.829856 18476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:45.830777 18476 log.cc:824] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:45.833072 18476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041: No bootstrap required, opened a new log
Aug 06 02:54:45 dist-test-slave-3kk6 krb5kdc[18351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:45.837503 18476 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2c879673a742ce8dd7cf35b3d09041" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43409 } }
I20260806 02:54:45.837868 18476 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:45.837965 18476 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2c879673a742ce8dd7cf35b3d09041, State: Initialized, Role: FOLLOWER
I20260806 02:54:45.838524 18476 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2c879673a742ce8dd7cf35b3d09041" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43409 } }
I20260806 02:54:45.838714 18476 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:45.838835 18476 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:45.838963 18476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:45.840222 18476 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2c879673a742ce8dd7cf35b3d09041" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43409 } }
I20260806 02:54:45.840658 18476 leader_election.cc:302] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae2c879673a742ce8dd7cf35b3d09041; no voters:
I20260806 02:54:45.841112 18476 leader_election.cc:288] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:45.841225 18484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:45.841449 18484 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [term 1 LEADER]: Becoming Leader. State: Replica: ae2c879673a742ce8dd7cf35b3d09041, State: Running, Role: LEADER
I20260806 02:54:45.841861 18484 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2c879673a742ce8dd7cf35b3d09041" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43409 } }
I20260806 02:54:45.842204 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:45.827479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.24.130.190:43409)
0806 02:54:45.828025 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:45.828041 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:45.828084 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:54:45.828833 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:45.828854 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:45.829221 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:45.829631 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:45.830565 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:45.831986 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:45.832954 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:45.833518 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:45.833704 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:45.836515 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:45.836522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:45.836538 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:45.836571 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:45.838603 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:45.839329 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:45.839334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:45.839339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:45.839468 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:45.839780 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:45.839783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:45.839784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:45.840215 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:45.840426 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:45.840726 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:54:45.841058 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":178,"threads_started":1}
I20260806 02:54:45.842551 18476 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:45.843225 18484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2c879673a742ce8dd7cf35b3d09041. Latest consensus state: current_term: 1 leader_uuid: "ae2c879673a742ce8dd7cf35b3d09041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2c879673a742ce8dd7cf35b3d09041" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43409 } } }
I20260806 02:54:45.843894 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae2c879673a742ce8dd7cf35b3d09041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2c879673a742ce8dd7cf35b3d09041" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 43409 } } }
I20260806 02:54:45.844028 18484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:45.844069 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:45.844594 18489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:45.848212 18489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:45.854815 18489 catalog_manager.cc:1347] Generated new cluster ID: e5d1264acb9b4823934e86ffa48921d8
I20260806 02:54:45.854918 18489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:45.895785 18489 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:45.896878 18489 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:45.904112 18489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041: Generated new TSK 0
I20260806 02:54:45.904891 18489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:54:45.942108 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43409
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:46.064069 18509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:46.064368 18509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:46.064414 18509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:46.068396 18509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:46.068456 18509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:46.068516 18509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:46.073917 18509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:46.075093 18509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:46.076107 18509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:46.086069 18517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:46.086069 18521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:46.086143 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:46.086182 18509 server_base.cc:1034] running on GCE node
I20260806 02:54:46.086817 18509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:46.087498 18509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:46.088676 18509 hybrid_clock.cc:648] HybridClock initialized: now 1785984886088628 us; error 60 us; skew 500 ppm
Aug 06 02:54:46 dist-test-slave-3kk6 krb5kdc[18351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:46.091681 18509 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:54:46.092628 18509 webserver.cc:460] Webserver started at http://127.24.130.129:42017/ using document root <none> and password file <none>
I20260806 02:54:46.093245 18509 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:46.093309 18509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:46.093567 18509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:46.095811 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "ece6de45a49240df934820eedd909d16"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "1f450d181881e3ef1a53afa86dc8104c"
server_key_iv: "cdc7ecf78210f77591fa981020002ad7"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.096424 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "ece6de45a49240df934820eedd909d16"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "1f450d181881e3ef1a53afa86dc8104c"
server_key_iv: "cdc7ecf78210f77591fa981020002ad7"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.101102 18509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:54:46.104377 18530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.105614 18509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:54:46.105762 18509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "ece6de45a49240df934820eedd909d16"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "1f450d181881e3ef1a53afa86dc8104c"
server_key_iv: "cdc7ecf78210f77591fa981020002ad7"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.105896 18509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:46.127105 18509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:46.130363 18509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:46.130594 18509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:46.131295 18509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:46.132200 18509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:46.132273 18509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.132344 18509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:46.132393 18509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.141559 18509 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:38059
I20260806 02:54:46.141584 18682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:38059 every 8 connection(s)
I20260806 02:54:46.142500 18509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:46.151340 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 18509
I20260806 02:54:46.151510 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:46.151909 25098 external_mini_cluster.cc:1428] Setting key 356f273232abc9c53079858247e23a66
Aug 06 02:54:46 dist-test-slave-3kk6 krb5kdc[18351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:46.157385 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.144480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:45777 (local address 127.24.130.190:43409)
0806 02:54:46.144649 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.144653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.145572 (+ 919us) server_negotiation.cc:407] Connection header received
0806 02:54:46.147050 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.147053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.147116 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.147228 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:46.148453 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.149011 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.150211 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.150455 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.150577 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.153594 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:46.153599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:46.153606 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:46.153631 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:46.155202 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.155735 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.155740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.155742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.155828 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.156197 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.156202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.156204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.156399 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:46.156513 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.157033 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.157180 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:54:46.158282 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.144843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43409 (local address 127.24.130.129:45777)
0806 02:54:46.145404 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.145453 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.146818 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.147411 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.147418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.147806 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:46.148290 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.148303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.149148 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.149153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.150067 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.150077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.150623 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.150627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.150835 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.151519 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:46.151545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:46.153435 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:46.155329 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:46.155335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:46.155350 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:46.155624 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:46.155931 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:46.155934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:46.155936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:46.156097 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:46.156512 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:46.156518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:46.156783 (+ 265us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.156993 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.157340 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":120,"threads_started":1}
I20260806 02:54:46.159617 18684 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43409
I20260806 02:54:46.159905 18684 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:46.160632 18684 heartbeater.cc:511] Master 127.24.130.190:43409 requested a full tablet report, sending...
I20260806 02:54:46.162585 18416 ts_manager.cc:194] Registered new tserver with Master: ece6de45a49240df934820eedd909d16 (127.24.130.129:38059)
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:54:46.164295 18416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:45777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:54:46.181947 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43409
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:46.296901 18692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:46.297197 18692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:46.297269 18692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:46.301578 18692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:46.301654 18692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:46.301748 18692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:46.307338 18692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:46.308481 18692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:46.309621 18692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:54:46.320030 18692 server_base.cc:1034] running on GCE node
W20260806 02:54:46.319993 18703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:46.319994 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:46.319984 18698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:46.320699 18692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:46.321365 18692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:46.322566 18692 hybrid_clock.cc:648] HybridClock initialized: now 1785984886322548 us; error 32 us; skew 500 ppm
Aug 06 02:54:46 dist-test-slave-3kk6 krb5kdc[18351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:46.325433 18692 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:54:46.326395 18692 webserver.cc:460] Webserver started at http://127.24.130.130:41033/ using document root <none> and password file <none>
I20260806 02:54:46.327049 18692 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:46.327147 18692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:46.327394 18692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:46.329545 18692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "85e6a0e234784659a74734a04e9c467b"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "2a0628d3e2bb96219eaf7592860467c0"
server_key_iv: "14c7fac92702aa942aee84f2e5afd9ef"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.330145 18692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "85e6a0e234784659a74734a04e9c467b"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "2a0628d3e2bb96219eaf7592860467c0"
server_key_iv: "14c7fac92702aa942aee84f2e5afd9ef"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.334069 18692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:54:46.336648 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.337875 18692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:54:46.338038 18692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "85e6a0e234784659a74734a04e9c467b"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "2a0628d3e2bb96219eaf7592860467c0"
server_key_iv: "14c7fac92702aa942aee84f2e5afd9ef"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.338160 18692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:46.355537 18692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:46.358922 18692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:46.359148 18692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:46.359795 18692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:46.360675 18692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:46.360749 18692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.360824 18692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:46.360875 18692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.370229 18692 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:35101
I20260806 02:54:46.370265 18866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:35101 every 8 connection(s)
I20260806 02:54:46.371272 18692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:46.379348 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 18692
I20260806 02:54:46.379524 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:46.379905 25098 external_mini_cluster.cc:1428] Setting key 002c02f9c891bc0bb4855fb8ac2e4dea
Aug 06 02:54:46 dist-test-slave-3kk6 krb5kdc[18351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:46.386633 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.373215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:51615 (local address 127.24.130.190:43409)
0806 02:54:46.373369 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.373373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.374142 (+ 769us) server_negotiation.cc:407] Connection header received
0806 02:54:46.375592 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.375596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.375661 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.375752 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:46.377034 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.377584 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.378793 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.379037 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.379172 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.382244 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:46.382250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:46.382256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:46.382291 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:46.384121 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.384808 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.384810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.384812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.384900 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.385336 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.385339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.385340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.385512 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:46.385637 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.386234 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.386382 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:54:46.387465 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.373539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43409 (local address 127.24.130.130:51615)
0806 02:54:46.373986 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.374030 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.375362 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.375944 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.375951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.376319 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:46.376883 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.376897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.377719 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.377731 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.378615 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.378625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.379215 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.379219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.379419 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.380221 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:46.380237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:46.382054 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:46.384265 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:46.384274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:46.384292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:46.384703 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:46.385008 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:46.385013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:46.385016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:46.385232 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:46.385640 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:46.385645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:46.385962 (+ 317us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.386135 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.386439 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260806 02:54:46.388747 18867 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43409
I20260806 02:54:46.389039 18867 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:46.389703 18867 heartbeater.cc:511] Master 127.24.130.190:43409 requested a full tablet report, sending...
I20260806 02:54:46.390954 18416 ts_manager.cc:194] Registered new tserver with Master: 85e6a0e234784659a74734a04e9c467b (127.24.130.130:35101)
I20260806 02:54:46.391553 18416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:51615
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:54:46.413335 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43409
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:46.529592 18875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:46.529915 18875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:46.529969 18875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:46.534178 18875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:46.534271 18875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:46.534333 18875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:46.539984 18875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:43409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:46.541232 18875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:46.542466 18875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:46.553385 18880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:46.553385 18881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:46.553426 18883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:46.553725 18875 server_base.cc:1034] running on GCE node
I20260806 02:54:46.554229 18875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:46.555006 18875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:46.556187 18875 hybrid_clock.cc:648] HybridClock initialized: now 1785984886556168 us; error 30 us; skew 500 ppm
Aug 06 02:54:46 dist-test-slave-3kk6 krb5kdc[18351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:46.559211 18875 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:54:46.560163 18875 webserver.cc:460] Webserver started at http://127.24.130.131:42965/ using document root <none> and password file <none>
I20260806 02:54:46.560822 18875 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:46.560921 18875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:46.561167 18875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:46.563414 18875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "edf246af65954ed9840ac39ba5ba858e"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "93fab3a112bbb59777bc3fbe6cacc34e"
server_key_iv: "b874f00d46f7dd3d336752f2128275ba"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.564030 18875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "edf246af65954ed9840ac39ba5ba858e"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "93fab3a112bbb59777bc3fbe6cacc34e"
server_key_iv: "b874f00d46f7dd3d336752f2128275ba"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.567982 18875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:54:46.570492 18898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.571864 18875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:54:46.572013 18875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "edf246af65954ed9840ac39ba5ba858e"
format_stamp: "Formatted at 2026-08-06 02:54:46 on dist-test-slave-3kk6"
server_key: "93fab3a112bbb59777bc3fbe6cacc34e"
server_key_iv: "b874f00d46f7dd3d336752f2128275ba"
server_key_version: "encryptionkey@0"
I20260806 02:54:46.572139 18875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:46.587196 18875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:46.590161 18875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:46.590387 18875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:46.591035 18875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:46.592067 18875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:46.592137 18875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.592204 18875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:46.592258 18875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:46.601505 18875 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:37793
I20260806 02:54:46.601528 19058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:37793 every 8 connection(s)
I20260806 02:54:46.602447 18875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:46.611861 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 18875
I20260806 02:54:46.612051 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:46.612377 25098 external_mini_cluster.cc:1428] Setting key b9d0998b38919fbd5d9615944686e964
Aug 06 02:54:46 dist-test-slave-3kk6 krb5kdc[18351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984886, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:46.616914 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.604426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:56761 (local address 127.24.130.190:43409)
0806 02:54:46.604571 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.604575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.605436 (+ 861us) server_negotiation.cc:407] Connection header received
0806 02:54:46.606809 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.606845 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.606900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.606983 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:46.608132 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.608655 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.609908 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.610173 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.610277 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.613284 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:46.613289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:46.613293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:46.613323 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:46.615023 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.615515 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.615518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.615519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.615594 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.615897 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.615900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.615901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.616082 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:46.616161 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.616640 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.616783 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:54:46.617720 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.604803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:43409 (local address 127.24.130.131:56761)
0806 02:54:46.605264 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.605309 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.606523 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.607142 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.607148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.607508 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:46.607999 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.608011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.608766 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.608768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.609754 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.609764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.610319 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.610322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.610470 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.611166 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:46.611182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:46.613080 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:46.615129 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:46.615134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:46.615147 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:46.615426 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:46.615674 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:46.615676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:46.615677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:46.615812 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:46.616178 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:46.616183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:46.616433 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.616591 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.616856 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260806 02:54:46.618935 19059 heartbeater.cc:347] Connected to a master server at 127.24.130.190:43409
I20260806 02:54:46.619159 19059 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:46.619638 19059 heartbeater.cc:511] Master 127.24.130.190:43409 requested a full tablet report, sending...
I20260806 02:54:46.620643 18416 ts_manager.cc:194] Registered new tserver with Master: edf246af65954ed9840ac39ba5ba858e (127.24.130.131:37793)
I20260806 02:54:46.621232 18416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:56761
I20260806 02:54:46.626399 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:46.636255 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.628483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.24.130.190:43409)
0806 02:54:46.628617 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.628621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.628847 (+ 226us) server_negotiation.cc:407] Connection header received
0806 02:54:46.628942 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.628944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.628984 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.629046 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:46.629730 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.630241 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.631127 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.631340 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.631418 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.632982 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:46.632985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:46.632986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:46.633011 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:46.634501 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.635031 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.635034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.635036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.635111 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:46.635466 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:46.635468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:46.635469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:46.635673 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:46.635789 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.636043 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.636127 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:54:46.639410 18416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:46.642724 18416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:46.657894 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.652005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.24.130.130:35101)
0806 02:54:46.653403 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.653412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.653432 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:54:46.653494 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.653498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.653636 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.653805 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:46.654180 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.655073 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.656564 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.657151 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.657296 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.657458 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.657626 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.657726 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":216,"threads_started":1}
I20260806 02:54:46.657886 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.651858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:35101 (local address 127.0.0.1:45514)
0806 02:54:46.652237 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.652287 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.652420 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.653793 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.653799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.653821 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:46.654025 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.654036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.655219 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.655222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.656435 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.656444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.657287 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.657290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.657410 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.657451 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.657594 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.657725 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260806 02:54:46.658406 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.652126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:38059 (local address 127.0.0.1:42170)
0806 02:54:46.652760 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.652773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.652873 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.653838 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.653841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.653873 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:46.654149 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.654157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.655613 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.655619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.657132 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.657144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.657984 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.657987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.658086 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.658110 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.658183 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.658258 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":238,"spinlock_wait_cycles":13568,"thread_start_us":105,"threads_started":1}
I20260806 02:54:46.658450 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.652890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.24.130.129:38059)
0806 02:54:46.653284 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.653307 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.653336 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:54:46.653419 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.653424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.653641 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.654036 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:46.654246 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.655440 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.657284 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.657849 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.657997 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.658124 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.658220 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.658314 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":76,"threads_started":1}
I20260806 02:54:46.659912 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.652676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:37793 (local address 127.0.0.1:58980)
0806 02:54:46.653051 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.653065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.653144 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.653838 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.653841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.653875 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:46.654063 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.654070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.655639 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.655642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.657132 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.657144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.659431 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.659435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.659552 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.659587 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.659645 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.659771 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":206,"spinlock_wait_cycles":5248,"thread_start_us":104,"threads_started":1}
I20260806 02:54:46.660676 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.653005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.24.130.131:37793)
0806 02:54:46.653451 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.653462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.653481 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:54:46.653543 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.653547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.653689 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.654203 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:46.654360 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.655496 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.658317 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.659226 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.660256 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.660385 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.660441 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.660533 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":147,"threads_started":1}
I20260806 02:54:46.661520 18783 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb5a83b7504c5898a4b9250cc65053 (DEFAULT_TABLE table=test-table [id=f0a28aa888f24c3dbd3dff5f01c4d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:46.661821 18594 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb5a83b7504c5898a4b9250cc65053 (DEFAULT_TABLE table=test-table [id=f0a28aa888f24c3dbd3dff5f01c4d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:46.662627 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb5a83b7504c5898a4b9250cc65053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:46.662950 18594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb5a83b7504c5898a4b9250cc65053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:46.663442 18966 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb5a83b7504c5898a4b9250cc65053 (DEFAULT_TABLE table=test-table [id=f0a28aa888f24c3dbd3dff5f01c4d1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:46.664454 18966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb5a83b7504c5898a4b9250cc65053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:46.669802 19082 tablet_bootstrap.cc:492] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: Bootstrap starting.
I20260806 02:54:46.670900 19085 tablet_bootstrap.cc:492] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: Bootstrap starting.
I20260806 02:54:46.671799 19087 tablet_bootstrap.cc:492] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Bootstrap starting.
I20260806 02:54:46.672524 19082 tablet_bootstrap.cc:654] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:46.672968 19085 tablet_bootstrap.cc:654] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:46.673705 19085 log.cc:824] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:46.673802 19082 log.cc:824] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:46.674093 19087 tablet_bootstrap.cc:654] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:46.675361 19087 log.cc:824] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:46.676321 19082 tablet_bootstrap.cc:492] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: No bootstrap required, opened a new log
I20260806 02:54:46.676328 19085 tablet_bootstrap.cc:492] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: No bootstrap required, opened a new log
I20260806 02:54:46.676532 19085 ts_tablet_manager.cc:1397] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:54:46.676533 19082 ts_tablet_manager.cc:1397] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:54:46.677516 19087 tablet_bootstrap.cc:492] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: No bootstrap required, opened a new log
I20260806 02:54:46.677693 19087 ts_tablet_manager.cc:1397] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:54:46.680146 19085 raft_consensus.cc:348] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.680413 19085 raft_consensus.cc:374] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:46.680469 19085 raft_consensus.cc:729] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece6de45a49240df934820eedd909d16, State: Initialized, Role: FOLLOWER
I20260806 02:54:46.680465 19082 raft_consensus.cc:348] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.680718 19082 raft_consensus.cc:374] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:46.680764 19082 raft_consensus.cc:729] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e6a0e234784659a74734a04e9c467b, State: Initialized, Role: FOLLOWER
I20260806 02:54:46.680950 19085 consensus_queue.cc:260] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.681192 19082 consensus_queue.cc:260] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.681751 18684 heartbeater.cc:503] Master 127.24.130.190:43409 was elected leader, sending a full tablet report...
I20260806 02:54:46.682433 19085 ts_tablet_manager.cc:1428] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:54:46.682508 19082 ts_tablet_manager.cc:1428] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:54:46.681418 19087 raft_consensus.cc:348] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.682796 19087 raft_consensus.cc:374] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:46.682865 19087 raft_consensus.cc:729] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf246af65954ed9840ac39ba5ba858e, State: Initialized, Role: FOLLOWER
I20260806 02:54:46.682880 18867 heartbeater.cc:503] Master 127.24.130.190:43409 was elected leader, sending a full tablet report...
I20260806 02:54:46.683277 19087 consensus_queue.cc:260] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.683916 19059 heartbeater.cc:503] Master 127.24.130.190:43409 was elected leader, sending a full tablet report...
I20260806 02:54:46.684191 19087 ts_tablet_manager.cc:1428] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:54:46.731537 19094 raft_consensus.cc:482] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:46.731827 19094 raft_consensus.cc:504] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.733098 19094 leader_election.cc:288] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf246af65954ed9840ac39ba5ba858e (127.24.130.131:37793), ece6de45a49240df934820eedd909d16 (127.24.130.129:38059)
I20260806 02:54:46.736848 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.733444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:54677 (local address 127.24.130.131:37793)
0806 02:54:46.733570 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.733574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.733590 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:54:46.733733 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.733741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.733793 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.733888 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:46.734187 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.734769 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.735831 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.736244 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.736395 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.736452 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.736633 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.736704 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:54:46.736855 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.733331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:37793 (local address 127.24.130.130:54677)
0806 02:54:46.733457 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.733475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.733578 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.733915 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.733918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.733941 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:46.734071 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.734076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.734887 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.734893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.735703 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.735712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.736388 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.736391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.736506 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.736525 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.736617 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.736696 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260806 02:54:46.737234 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.733526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:48859 (local address 127.24.130.129:38059)
0806 02:54:46.733692 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.733697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.734083 (+ 386us) server_negotiation.cc:407] Connection header received
0806 02:54:46.734219 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.734223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.734268 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.734354 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:46.734634 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.735145 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.736211 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.736614 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.736756 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.736807 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.737046 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.737127 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:54:46.737601 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.733352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:38059 (local address 127.24.130.130:48859)
0806 02:54:46.733984 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.733996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:46.734115 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:46.734365 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:46.734369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:46.734384 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:46.734517 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.734522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.735269 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.735275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.736069 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:46.736083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.736750 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.736753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:46.736859 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.736883 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:54:46.737402 (+ 519us) client_negotiation.cc:240] Negotiation successful
0806 02:54:46.737479 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":276,"thread_start_us":185,"threads_started":1}
I20260806 02:54:46.737632 18992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb5a83b7504c5898a4b9250cc65053" candidate_uuid: "85e6a0e234784659a74734a04e9c467b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf246af65954ed9840ac39ba5ba858e" is_pre_election: true
I20260806 02:54:46.738021 18992 raft_consensus.cc:2454] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e6a0e234784659a74734a04e9c467b in term 0.
I20260806 02:54:46.738340 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb5a83b7504c5898a4b9250cc65053" candidate_uuid: "85e6a0e234784659a74734a04e9c467b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece6de45a49240df934820eedd909d16" is_pre_election: true
I20260806 02:54:46.738696 18623 raft_consensus.cc:2454] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e6a0e234784659a74734a04e9c467b in term 0.
I20260806 02:54:46.738607 18716 leader_election.cc:302] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e6a0e234784659a74734a04e9c467b, edf246af65954ed9840ac39ba5ba858e; no voters:
I20260806 02:54:46.738994 19094 raft_consensus.cc:2781] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:46.739102 19094 raft_consensus.cc:482] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:46.739166 19094 raft_consensus.cc:3037] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:46.740281 19094 raft_consensus.cc:504] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.740661 19094 leader_election.cc:288] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [CANDIDATE]: Term 1 election: Requested vote from peers edf246af65954ed9840ac39ba5ba858e (127.24.130.131:37793), ece6de45a49240df934820eedd909d16 (127.24.130.129:38059)
I20260806 02:54:46.741070 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb5a83b7504c5898a4b9250cc65053" candidate_uuid: "85e6a0e234784659a74734a04e9c467b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece6de45a49240df934820eedd909d16"
I20260806 02:54:46.741187 18623 raft_consensus.cc:3037] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:46.742048 18992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb5a83b7504c5898a4b9250cc65053" candidate_uuid: "85e6a0e234784659a74734a04e9c467b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf246af65954ed9840ac39ba5ba858e"
I20260806 02:54:46.742208 18992 raft_consensus.cc:3037] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:46.742560 18623 raft_consensus.cc:2454] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e6a0e234784659a74734a04e9c467b in term 1.
I20260806 02:54:46.742975 18718 leader_election.cc:302] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e6a0e234784659a74734a04e9c467b, ece6de45a49240df934820eedd909d16; no voters:
I20260806 02:54:46.743184 19094 raft_consensus.cc:2781] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:46.743453 19094 raft_consensus.cc:686] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 1 LEADER]: Becoming Leader. State: Replica: 85e6a0e234784659a74734a04e9c467b, State: Running, Role: LEADER
I20260806 02:54:46.743748 18992 raft_consensus.cc:2454] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e6a0e234784659a74734a04e9c467b in term 1.
I20260806 02:54:46.743870 19094 consensus_queue.cc:237] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } }
I20260806 02:54:46.747434 18413 catalog_manager.cc:5495] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e6a0e234784659a74734a04e9c467b (127.24.130.130). New cstate: current_term: 1 leader_uuid: "85e6a0e234784659a74734a04e9c467b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e6a0e234784659a74734a04e9c467b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 } health_report { overall_health: UNKNOWN } } }
I20260806 02:54:46.760604 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.756608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.24.130.130:35101)
0806 02:54:46.756763 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:46.756768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:46.756784 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:54:46.756948 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:46.756951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:46.757010 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:46.757133 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:46.757407 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.757935 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.758921 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:46.759147 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:46.759269 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:46.759633 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:46.760074 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:46.760248 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:46.760369 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 02:54:46.760416 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:54:46.766949 18623 raft_consensus.cc:1261] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16 [term 1 FOLLOWER]: Refusing update from remote peer 85e6a0e234784659a74734a04e9c467b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:46.766948 18992 raft_consensus.cc:1261] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 1 FOLLOWER]: Refusing update from remote peer 85e6a0e234784659a74734a04e9c467b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:46.767735 19103 consensus_queue.cc:1035] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece6de45a49240df934820eedd909d16" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 38059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:46.767982 19094 consensus_queue.cc:1035] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf246af65954ed9840ac39ba5ba858e" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:46.776487 19109 mvcc.cc:205] Tried to move back new op lower bound from 7315394096189353984 to 7315394096105738240. Current Snapshot: MvccSnapshot[applied={T|T < 7315394096189353984}]
I20260806 02:54:46.776757 19108 mvcc.cc:205] Tried to move back new op lower bound from 7315394096189353984 to 7315394096105738240. Current Snapshot: MvccSnapshot[applied={T|T < 7315394096189353984}]
I20260806 02:54:46.785663 18413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:46.785849 18413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:46.788475 18413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae2c879673a742ce8dd7cf35b3d09041: Sending DeleteTablet for 3 replicas of tablet d2bb5a83b7504c5898a4b9250cc65053
I20260806 02:54:46.789572 18594 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb5a83b7504c5898a4b9250cc65053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:46 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:42170
I20260806 02:54:46.789588 18783 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb5a83b7504c5898a4b9250cc65053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:46 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:45514
I20260806 02:54:46.789567 18966 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb5a83b7504c5898a4b9250cc65053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:46 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:58980
I20260806 02:54:46.790149 19120 tablet_replica.cc:330] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: stopping tablet replica
I20260806 02:54:46.790337 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 18509
I20260806 02:54:46.790560 19121 tablet_replica.cc:330] T d2bb5a83b7504c5898a4b9250cc65053 P ece6de45a49240df934820eedd909d16: stopping tablet replica
I20260806 02:54:46.790777 19123 tablet_replica.cc:330] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: stopping tablet replica
I20260806 02:54:46.791163 19123 raft_consensus.cc:2229] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:46.793494 19123 raft_consensus.cc:2258] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:46.794387 19120 raft_consensus.cc:2229] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:46.794747 19120 raft_consensus.cc:2258] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:46.797748 19120 ts_tablet_manager.cc:1905] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:46.801739 19120 ts_tablet_manager.cc:1918] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:46.801889 19120 log.cc:1196] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/d2bb5a83b7504c5898a4b9250cc65053
I20260806 02:54:46.802285 19120 ts_tablet_manager.cc:1939] T d2bb5a83b7504c5898a4b9250cc65053 P edf246af65954ed9840ac39ba5ba858e: Deleting consensus metadata
I20260806 02:54:46.803395 19123 ts_tablet_manager.cc:1905] T d2bb5a83b7504c5898a4b9250cc65053 P 85e6a0e234784659a74734a04e9c467b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:46.804033 18393 catalog_manager.cc:4841] TS edf246af65954ed9840ac39ba5ba858e (127.24.130.131:37793): tablet d2bb5a83b7504c5898a4b9250cc65053 (table test-table [id=f0a28aa888f24c3dbd3dff5f01c4d1e5]) successfully deleted
W20260806 02:54:46.804418 18395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:38059 (error 108)
I20260806 02:54:46.804502 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 18692
I20260806 02:54:46.827009 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.826520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:38059 (local address 127.0.0.1:42174)
0806 02:54:46.826845 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.826921 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:38059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:54:46.827260 18395 catalog_manager.cc:4584] TS ece6de45a49240df934820eedd909d16 (127.24.130.129:38059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2bb5a83b7504c5898a4b9250cc65053: Network error: Client connection negotiation failed: client connection to 127.24.130.129:38059: connect: Connection refused (error 111)
I20260806 02:54:46.833084 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 18875
I20260806 02:54:46.837657 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:46.833485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:35101 (local address 127.0.0.1:45538)
0806 02:54:46.837442 (+ 3957us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:46.837531 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.130:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3826}
I20260806 02:54:46.849014 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 18370
2026-08-06T02:54:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2604 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T02:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:46Z Disabled control of system clock
I20260806 02:54:46.890156 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39927
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:54:47.010592 19129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:47.010955 19129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:47.010994 19129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:47.015340 19129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:47.015432 19129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:47.015448 19129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:47.015467 19129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:47.015483 19129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:47.020725 19129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:47.021941 19129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:47.023404 19129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:47.032833 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:47.032847 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:47.032913 19129 server_base.cc:1034] running on GCE node
W20260806 02:54:47.032853 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:47.033725 19129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:47.035074 19129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:47.036289 19129 hybrid_clock.cc:648] HybridClock initialized: now 1785984887036271 us; error 37 us; skew 500 ppm
I20260806 02:54:47.038403 19129 webserver.cc:460] Webserver started at http://127.24.130.190:33499/ using document root <none> and password file <none>
I20260806 02:54:47.039112 19129 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:47.039222 19129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:47.039487 19129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:47.041786 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "71fe51601e97405cb35ab6ded7f53b0c"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "4d515bb11378399697997226db6333ad"
server_key_iv: "71f0110ca19c70c18637c2d58fc5d537"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.042411 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "71fe51601e97405cb35ab6ded7f53b0c"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "4d515bb11378399697997226db6333ad"
server_key_iv: "71f0110ca19c70c18637c2d58fc5d537"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.046443 19129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:54:47.049470 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.050823 19129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:47.050973 19129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "71fe51601e97405cb35ab6ded7f53b0c"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "4d515bb11378399697997226db6333ad"
server_key_iv: "71f0110ca19c70c18637c2d58fc5d537"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.051103 19129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:47.075779 19129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:47.076648 19129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:47.076893 19129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:47.084798 19129 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:39927
I20260806 02:54:47.084798 19232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:39927 every 8 connection(s)
I20260806 02:54:47.085878 19129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:47.088102 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 19129
I20260806 02:54:47.088258 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:47.088582 25098 external_mini_cluster.cc:1428] Setting key 677b719b395213bcbdb3580cf1491987
I20260806 02:54:47.089540 19233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:47.096825 19233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c: Bootstrap starting.
I20260806 02:54:47.099087 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.090080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.24.130.190:39927)
0806 02:54:47.090810 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.090828 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.090868 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:54:47.091558 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.091575 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.091992 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.092296 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:47.092681 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.093673 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.094969 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.095622 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.095757 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.096797 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:47.096802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:47.096817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:47.097331 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:47.097395 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:47.097418 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:47.097543 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.097737 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.098031 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":160,"threads_started":1}
I20260806 02:54:47.099756 19233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:47.100629 19233 log.cc:824] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:47.103097 19233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c: No bootstrap required, opened a new log
I20260806 02:54:47.107125 19233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fe51601e97405cb35ab6ded7f53b0c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39927 } }
I20260806 02:54:47.107399 19233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:47.107439 19233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71fe51601e97405cb35ab6ded7f53b0c, State: Initialized, Role: FOLLOWER
I20260806 02:54:47.107972 19233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fe51601e97405cb35ab6ded7f53b0c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39927 } }
I20260806 02:54:47.108135 19233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:47.108227 19233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:47.108355 19233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:47.109421 19233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fe51601e97405cb35ab6ded7f53b0c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39927 } }
I20260806 02:54:47.109807 19233 leader_election.cc:302] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71fe51601e97405cb35ab6ded7f53b0c; no voters:
I20260806 02:54:47.110167 19233 leader_election.cc:288] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:47.110332 19241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:47.110648 19241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [term 1 LEADER]: Becoming Leader. State: Replica: 71fe51601e97405cb35ab6ded7f53b0c, State: Running, Role: LEADER
I20260806 02:54:47.111013 19241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fe51601e97405cb35ab6ded7f53b0c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39927 } }
I20260806 02:54:47.111603 19233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:47.112802 19242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71fe51601e97405cb35ab6ded7f53b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fe51601e97405cb35ab6ded7f53b0c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39927 } } }
I20260806 02:54:47.112944 19242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:47.113471 19257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:47.113417 19244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71fe51601e97405cb35ab6ded7f53b0c. Latest consensus state: current_term: 1 leader_uuid: "71fe51601e97405cb35ab6ded7f53b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fe51601e97405cb35ab6ded7f53b0c" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39927 } } }
I20260806 02:54:47.113583 19244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:47.117349 19257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:47.124192 19257 catalog_manager.cc:1347] Generated new cluster ID: a8d5b9357d994a6b85b5979218bb2b3a
I20260806 02:54:47.124301 19257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:47.145200 19257 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:47.146600 19257 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:47.152160 19257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c: Generated new TSK 0
I20260806 02:54:47.153012 19257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:47.164672 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39927
--builtin_ntp_servers=127.24.130.148:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:54:47.286355 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:47.286767 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:47.286826 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:47.291077 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:47.291155 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:47.291216 19271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:47.296239 19271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:47.297420 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:47.298902 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:47.309006 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:47.309073 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:47.309072 19283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:47.309476 19271 server_base.cc:1034] running on GCE node
I20260806 02:54:47.309911 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:47.310575 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:47.311770 19271 hybrid_clock.cc:648] HybridClock initialized: now 1785984887311752 us; error 39 us; skew 500 ppm
I20260806 02:54:47.314193 19271 webserver.cc:460] Webserver started at http://127.24.130.129:41661/ using document root <none> and password file <none>
I20260806 02:54:47.314913 19271 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:47.314986 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:47.315228 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:47.317588 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "637a1c67763f4d23b84640034b2b9102"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "771b22efd04df5e84860cb4f8b1300ad"
server_key_iv: "b737a20b57bc800c40e213b4dfc0b0e0"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.318261 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "637a1c67763f4d23b84640034b2b9102"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "771b22efd04df5e84860cb4f8b1300ad"
server_key_iv: "b737a20b57bc800c40e213b4dfc0b0e0"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.322468 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:54:47.325359 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.326875 19271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:47.327028 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "637a1c67763f4d23b84640034b2b9102"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "771b22efd04df5e84860cb4f8b1300ad"
server_key_iv: "b737a20b57bc800c40e213b4dfc0b0e0"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.327160 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:47.340368 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:47.341208 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:47.341445 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:47.342140 19271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:47.343256 19271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:47.343333 19271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.343407 19271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:47.343462 19271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.353165 19271 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:42657
I20260806 02:54:47.353186 19442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:42657 every 8 connection(s)
I20260806 02:54:47.354254 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:47.363950 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.356279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:56535 (local address 127.24.130.190:39927)
0806 02:54:47.356463 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.356468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.357576 (+ 1108us) server_negotiation.cc:407] Connection header received
0806 02:54:47.359004 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.359007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.359065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.359155 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:47.360071 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.360644 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.361825 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.362050 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.362135 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.363413 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:47.363417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:47.363419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:47.363433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:47.363454 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:47.363462 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:47.363533 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.363677 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.363759 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:54:47.364223 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 19271
I20260806 02:54:47.364382 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:47.364727 25098 external_mini_cluster.cc:1428] Setting key 5d3108c5fa67dfc2624ae165a1392a87
I20260806 02:54:47.364926 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.356734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39927 (local address 127.24.130.129:56535)
0806 02:54:47.357415 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:47.357457 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:47.358772 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:47.359319 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:47.359325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:47.359418 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:47.359916 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.359928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.360769 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.360772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.361687 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.361697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.362193 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.362210 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.362367 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.363124 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:47.363126 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:47.363169 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:47.363238 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:47.363546 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:47.363553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:47.363557 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:54:47.363657 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:54:47.363895 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260806 02:54:47.366473 19443 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39927
I20260806 02:54:47.366791 19443 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:47.367009 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39927
--builtin_ntp_servers=127.24.130.148:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:54:47.367412 19443 heartbeater.cc:511] Master 127.24.130.190:39927 requested a full tablet report, sending...
I20260806 02:54:47.369210 19179 ts_manager.cc:194] Registered new tserver with Master: 637a1c67763f4d23b84640034b2b9102 (127.24.130.129:42657)
I20260806 02:54:47.370854 19179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.129:56535
W20260806 02:54:47.488005 19446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:47.488327 19446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:47.488369 19446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:47.492797 19446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:47.492882 19446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:47.492941 19446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:47.498119 19446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:47.499451 19446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:47.501014 19446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:47.511685 19454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:47.511816 19456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:47.512040 19446 server_base.cc:1034] running on GCE node
W20260806 02:54:47.511633 19453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:47.512616 19446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:47.513430 19446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:47.514595 19446 hybrid_clock.cc:648] HybridClock initialized: now 1785984887514563 us; error 48 us; skew 500 ppm
I20260806 02:54:47.516700 19446 webserver.cc:460] Webserver started at http://127.24.130.130:34029/ using document root <none> and password file <none>
I20260806 02:54:47.517326 19446 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:47.517397 19446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:47.517634 19446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:47.519933 19446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "ccdabf4c519c4ea6847946859873ce70"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "960d2e517886bb118974c8c849f18e4f"
server_key_iv: "47a490bd83988ab37b8f2ee131d107b0"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.520553 19446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "ccdabf4c519c4ea6847946859873ce70"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "960d2e517886bb118974c8c849f18e4f"
server_key_iv: "47a490bd83988ab37b8f2ee131d107b0"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.524616 19446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:54:47.527312 19465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.528649 19446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:47.528795 19446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "ccdabf4c519c4ea6847946859873ce70"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "960d2e517886bb118974c8c849f18e4f"
server_key_iv: "47a490bd83988ab37b8f2ee131d107b0"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.528937 19446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:47.543876 19446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:47.544708 19446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:47.544957 19446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:47.545641 19446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:47.546907 19446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:47.546994 19446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.547106 19446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:47.547137 19446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.557644 19446 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:45281
I20260806 02:54:47.557663 19614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:45281 every 8 connection(s)
I20260806 02:54:47.558806 19446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:47.567092 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 19446
I20260806 02:54:47.567265 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:47.567591 25098 external_mini_cluster.cc:1428] Setting key bc27047b52ac913ba35ee2e263dba465
I20260806 02:54:47.569483 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.560838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:56339 (local address 127.24.130.190:39927)
0806 02:54:47.561028 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.561032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.562256 (+ 1224us) server_negotiation.cc:407] Connection header received
0806 02:54:47.564227 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.564231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.564308 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.564428 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:47.565429 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.565976 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.567301 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.567561 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.567685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.568940 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:47.568945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:47.568948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:47.568968 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:47.568989 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:47.568997 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:47.569080 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.569259 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.569328 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:54:47.569868 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39927
--builtin_ntp_servers=127.24.130.148:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:54:47.570536 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.561278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39927 (local address 127.24.130.130:56339)
0806 02:54:47.562036 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:47.562084 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:47.563931 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:47.564631 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:47.564638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:47.564731 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:47.565265 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.565278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.566128 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.566131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.567160 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.567171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.567739 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.567743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.567926 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.568644 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:47.568646 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:47.568691 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:47.568769 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:47.569084 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:47.569096 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:47.569103 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:54:47.569246 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:54:47.569575 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":128,"threads_started":1}
I20260806 02:54:47.571810 19615 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39927
I20260806 02:54:47.572108 19615 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:47.572688 19615 heartbeater.cc:511] Master 127.24.130.190:39927 requested a full tablet report, sending...
I20260806 02:54:47.573939 19179 ts_manager.cc:194] Registered new tserver with Master: ccdabf4c519c4ea6847946859873ce70 (127.24.130.130:45281)
I20260806 02:54:47.574846 19179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.130:56339
W20260806 02:54:47.692761 19620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:47.693068 19620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:47.693109 19620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:47.697815 19620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:47.697911 19620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:47.697973 19620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:47.703693 19620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:47.704875 19620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:47.706453 19620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:47.717674 19627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:47.717677 19630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:47.717664 19626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:47.717728 19620 server_base.cc:1034] running on GCE node
I20260806 02:54:47.718472 19620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:47.719245 19620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:47.720458 19620 hybrid_clock.cc:648] HybridClock initialized: now 1785984887720441 us; error 34 us; skew 500 ppm
I20260806 02:54:47.723027 19620 webserver.cc:460] Webserver started at http://127.24.130.131:45139/ using document root <none> and password file <none>
I20260806 02:54:47.723888 19620 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:47.723996 19620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:47.724251 19620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:47.726864 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "7ac4ba67589d48d3afddce4311890f2f"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "a1dd4d35bbf6fdee804e3c69e34ca8af"
server_key_iv: "7c87d56748774aeb0fe30c0677f28283"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.727459 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac4ba67589d48d3afddce4311890f2f"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "a1dd4d35bbf6fdee804e3c69e34ca8af"
server_key_iv: "7c87d56748774aeb0fe30c0677f28283"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.731700 19620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:54:47.734623 19636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.736060 19620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:54:47.736168 19620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "7ac4ba67589d48d3afddce4311890f2f"
format_stamp: "Formatted at 2026-08-06 02:54:47 on dist-test-slave-3kk6"
server_key: "a1dd4d35bbf6fdee804e3c69e34ca8af"
server_key_iv: "7c87d56748774aeb0fe30c0677f28283"
server_key_version: "encryptionkey@0"
I20260806 02:54:47.736330 19620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:47.751425 19620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:47.752238 19620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:47.752478 19620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:47.753150 19620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:47.754177 19620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:47.754228 19620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.754271 19620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:47.754287 19620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:47.764235 19620 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:46119
I20260806 02:54:47.764266 19788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:46119 every 8 connection(s)
I20260806 02:54:47.765246 19620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:47.769269 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 19620
I20260806 02:54:47.769374 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:47.769667 25098 external_mini_cluster.cc:1428] Setting key 8bf7671f91dcd7c4aa641643c9668285
I20260806 02:54:47.775892 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.767315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:53887 (local address 127.24.130.190:39927)
0806 02:54:47.767494 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.767500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.768646 (+ 1146us) server_negotiation.cc:407] Connection header received
0806 02:54:47.770320 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.770324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.770381 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.770485 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:47.771840 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.772452 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.773696 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.773943 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.774034 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.775423 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:47.775426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:47.775428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:47.775442 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:47.775464 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:47.775472 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:47.775538 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.775684 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.775743 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:54:47.776798 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.767739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39927 (local address 127.24.130.131:53887)
0806 02:54:47.768482 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:47.768527 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:47.770079 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:47.770765 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:47.770779 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:47.770914 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:47.771642 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.771663 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.772604 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.772607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.773564 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.773575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.774116 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.774119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.774296 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.775137 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:47.775139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:47.775192 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:47.775266 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:47.775554 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:47.775561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:47.775566 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:54:47.775667 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:54:47.775920 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":133,"threads_started":1}
I20260806 02:54:47.777954 19790 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39927
I20260806 02:54:47.778172 19790 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:47.778837 19790 heartbeater.cc:511] Master 127.24.130.190:39927 requested a full tablet report, sending...
I20260806 02:54:47.779876 19179 ts_manager.cc:194] Registered new tserver with Master: 7ac4ba67589d48d3afddce4311890f2f (127.24.130.131:46119)
I20260806 02:54:47.780473 19179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.131:53887
I20260806 02:54:47.784310 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:47.790480 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.786497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.24.130.190:39927)
0806 02:54:47.786645 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.786691 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.786858 (+ 167us) server_negotiation.cc:407] Connection header received
0806 02:54:47.786949 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.786951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.786996 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.787071 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:47.787346 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.787880 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.788724 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.788949 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.789026 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.789994 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:47.789997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:47.789999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:47.790011 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:47.790027 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:47.790034 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:47.790129 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.790335 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.790367 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:54:47.793821 19179 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:47.797065 19179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:47.813329 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.807635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.24.130.130:45281)
0806 02:54:47.808424 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.808434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.808454 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:54:47.808517 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.808520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.808669 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.808845 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:47.809327 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.810483 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.811935 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.812592 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.812785 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.812991 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.813057 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.813153 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":152,"threads_started":1}
I20260806 02:54:47.813333 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.806705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:45281 (local address 127.0.0.1:37882)
0806 02:54:47.807882 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:47.807895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:47.807979 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:47.808836 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:47.808842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:47.808877 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:47.809172 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.809178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.810606 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.810609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.811794 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.811803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.812749 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.812754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.812872 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.812912 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:54:47.813031 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:54:47.813157 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":86,"spinlock_wait_cycles":1920,"thread_start_us":111,"threads_started":1}
I20260806 02:54:47.813761 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.806571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42657 (local address 127.0.0.1:37394)
0806 02:54:47.807420 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:47.807477 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:47.807607 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:47.808849 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:47.808852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:47.808881 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:47.809167 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.809178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.811138 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.811142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.812113 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.812125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.813211 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.813214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.813452 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.813479 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:54:47.813543 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:54:47.813609 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":167,"spinlock_wait_cycles":1280,"thread_start_us":146,"threads_started":1}
I20260806 02:54:47.814322 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.807748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:46119 (local address 127.0.0.1:57454)
0806 02:54:47.808127 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:47.808141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:47.808233 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:47.808985 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:47.808989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:47.809004 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:47.809163 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.809171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.810307 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.810312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.811762 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:47.811774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.813197 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.813201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:47.813677 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.814039 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 02:54:47.814108 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:54:47.814190 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":347,"thread_start_us":98,"threads_started":1}
I20260806 02:54:47.814882 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.806725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.24.130.129:42657)
0806 02:54:47.808302 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.808316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.808350 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:54:47.808437 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.808444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.808662 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.809071 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:47.809327 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.811007 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.812276 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.813079 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.814198 (+ 1119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.814376 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.814565 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.814694 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":156,"threads_started":1}
I20260806 02:54:47.815096 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:47.807944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.24.130.131:46119)
0806 02:54:47.808424 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:47.808434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:47.808453 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:54:47.808516 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:47.808520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:47.808662 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:47.808829 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:47.809274 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.810181 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.811911 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:47.812819 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:47.812947 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:47.813126 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:47.814802 (+ 1676us) server_negotiation.cc:299] Negotiation successful
0806 02:54:47.814917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":158,"threads_started":1}
I20260806 02:54:47.816956 19523 tablet_service.cc:1467] Processing CreateTablet for tablet 26b75b66d6ae44e495f0ff0f93bd6ade (DEFAULT_TABLE table=test-table [id=ead1f64fbd60470abe713ff022f30e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:47.817375 19352 tablet_service.cc:1467] Processing CreateTablet for tablet 26b75b66d6ae44e495f0ff0f93bd6ade (DEFAULT_TABLE table=test-table [id=ead1f64fbd60470abe713ff022f30e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:47.817786 19696 tablet_service.cc:1467] Processing CreateTablet for tablet 26b75b66d6ae44e495f0ff0f93bd6ade (DEFAULT_TABLE table=test-table [id=ead1f64fbd60470abe713ff022f30e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:47.818197 19523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b75b66d6ae44e495f0ff0f93bd6ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:47.818356 19352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b75b66d6ae44e495f0ff0f93bd6ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:47.818908 19696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b75b66d6ae44e495f0ff0f93bd6ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:47.825026 19816 tablet_bootstrap.cc:492] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: Bootstrap starting.
I20260806 02:54:47.826448 19818 tablet_bootstrap.cc:492] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Bootstrap starting.
I20260806 02:54:47.827001 19819 tablet_bootstrap.cc:492] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Bootstrap starting.
I20260806 02:54:47.827509 19816 tablet_bootstrap.cc:654] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:47.828492 19816 log.cc:824] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:47.828802 19818 tablet_bootstrap.cc:654] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:47.829331 19819 tablet_bootstrap.cc:654] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:47.829849 19818 log.cc:824] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:47.830176 19819 log.cc:824] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:47.830994 19816 tablet_bootstrap.cc:492] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: No bootstrap required, opened a new log
I20260806 02:54:47.831249 19816 ts_tablet_manager.cc:1397] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:54:47.832056 19818 tablet_bootstrap.cc:492] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: No bootstrap required, opened a new log
I20260806 02:54:47.832212 19819 tablet_bootstrap.cc:492] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: No bootstrap required, opened a new log
I20260806 02:54:47.832247 19818 ts_tablet_manager.cc:1397] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:54:47.832384 19819 ts_tablet_manager.cc:1397] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:54:47.835937 19816 raft_consensus.cc:348] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:47.836213 19816 raft_consensus.cc:374] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:47.836284 19816 raft_consensus.cc:729] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637a1c67763f4d23b84640034b2b9102, State: Initialized, Role: FOLLOWER
I20260806 02:54:47.836778 19816 consensus_queue.cc:260] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:47.836678 19818 raft_consensus.cc:348] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:47.836776 19819 raft_consensus.cc:348] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:47.836935 19818 raft_consensus.cc:374] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:47.836992 19818 raft_consensus.cc:729] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccdabf4c519c4ea6847946859873ce70, State: Initialized, Role: FOLLOWER
I20260806 02:54:47.837000 19819 raft_consensus.cc:374] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:47.837046 19819 raft_consensus.cc:729] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac4ba67589d48d3afddce4311890f2f, State: Initialized, Role: FOLLOWER
I20260806 02:54:47.837443 19818 consensus_queue.cc:260] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:47.837443 19819 consensus_queue.cc:260] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:47.837860 19816 ts_tablet_manager.cc:1428] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:54:47.838274 19443 heartbeater.cc:503] Master 127.24.130.190:39927 was elected leader, sending a full tablet report...
I20260806 02:54:47.838406 19818 ts_tablet_manager.cc:1428] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:54:47.838406 19819 ts_tablet_manager.cc:1428] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:54:47.838788 19615 heartbeater.cc:503] Master 127.24.130.190:39927 was elected leader, sending a full tablet report...
I20260806 02:54:47.839751 19790 heartbeater.cc:503] Master 127.24.130.190:39927 was elected leader, sending a full tablet report...
W20260806 02:54:47.855751 19444 tablet.cc:2369] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:54:48.016374 19791 tablet.cc:2369] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:48.039296 19825 raft_consensus.cc:482] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:48.039630 19825 raft_consensus.cc:504] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:48.040961 19825 leader_election.cc:288] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 637a1c67763f4d23b84640034b2b9102 (127.24.130.129:42657), 7ac4ba67589d48d3afddce4311890f2f (127.24.130.131:46119)
I20260806 02:54:48.046094 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:48.041376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:34053 (local address 127.24.130.129:42657)
0806 02:54:48.041509 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:48.041514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:48.041536 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:54:48.041723 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:48.041726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:48.041789 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:48.041896 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:48.042250 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.043101 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.044788 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.045405 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.045540 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:48.045592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:48.045873 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 02:54:48.045947 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:54:48.046094 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:48.041376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:37985 (local address 127.24.130.131:46119)
0806 02:54:48.041509 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:48.041514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:48.042055 (+ 541us) server_negotiation.cc:407] Connection header received
0806 02:54:48.042158 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:48.042160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:48.042231 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:48.042328 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:48.042563 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.043285 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.044788 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.045405 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.045540 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:48.045592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:48.045873 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 02:54:48.045948 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:54:48.046139 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:48.041294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42657 (local address 127.24.130.130:34053)
0806 02:54:48.041478 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:48.041495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:48.041587 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:48.041938 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:48.041941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:48.041964 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:48.042112 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:48.042121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.043223 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.043226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:48.044652 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:48.044664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.045544 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.045548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:48.045689 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:48.045756 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 02:54:48.045841 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:54:48.045974 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":176}
I20260806 02:54:48.046149 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:48.041294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:46119 (local address 127.24.130.130:37985)
0806 02:54:48.041949 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:48.041963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:48.042054 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:48.042358 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:48.042360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:48.042375 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:48.042457 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:48.042461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.043417 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.043420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:48.044653 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:48.044664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.045544 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.045548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:48.045673 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:48.045755 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:54:48.045843 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:54:48.045974 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":327,"thread_start_us":136,"threads_started":1}
I20260806 02:54:48.047092 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b75b66d6ae44e495f0ff0f93bd6ade" candidate_uuid: "ccdabf4c519c4ea6847946859873ce70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637a1c67763f4d23b84640034b2b9102" is_pre_election: true
I20260806 02:54:48.047098 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b75b66d6ae44e495f0ff0f93bd6ade" candidate_uuid: "ccdabf4c519c4ea6847946859873ce70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac4ba67589d48d3afddce4311890f2f" is_pre_election: true
I20260806 02:54:48.047469 19378 raft_consensus.cc:2454] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccdabf4c519c4ea6847946859873ce70 in term 0.
I20260806 02:54:48.047469 19721 raft_consensus.cc:2454] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccdabf4c519c4ea6847946859873ce70 in term 0.
I20260806 02:54:48.048029 19466 leader_election.cc:302] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637a1c67763f4d23b84640034b2b9102, ccdabf4c519c4ea6847946859873ce70; no voters:
I20260806 02:54:48.048337 19825 raft_consensus.cc:2781] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:48.048445 19825 raft_consensus.cc:482] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:48.048528 19825 raft_consensus.cc:3037] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:48.049968 19825 raft_consensus.cc:504] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:48.050395 19825 leader_election.cc:288] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [CANDIDATE]: Term 1 election: Requested vote from peers 637a1c67763f4d23b84640034b2b9102 (127.24.130.129:42657), 7ac4ba67589d48d3afddce4311890f2f (127.24.130.131:46119)
I20260806 02:54:48.050873 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b75b66d6ae44e495f0ff0f93bd6ade" candidate_uuid: "ccdabf4c519c4ea6847946859873ce70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac4ba67589d48d3afddce4311890f2f"
I20260806 02:54:48.050997 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b75b66d6ae44e495f0ff0f93bd6ade" candidate_uuid: "ccdabf4c519c4ea6847946859873ce70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637a1c67763f4d23b84640034b2b9102"
I20260806 02:54:48.051012 19721 raft_consensus.cc:3037] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:48.051164 19378 raft_consensus.cc:3037] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:48.052229 19721 raft_consensus.cc:2454] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccdabf4c519c4ea6847946859873ce70 in term 1.
I20260806 02:54:48.052239 19378 raft_consensus.cc:2454] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccdabf4c519c4ea6847946859873ce70 in term 1.
I20260806 02:54:48.052588 19470 leader_election.cc:302] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac4ba67589d48d3afddce4311890f2f, ccdabf4c519c4ea6847946859873ce70; no voters:
I20260806 02:54:48.052789 19825 raft_consensus.cc:2781] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:48.053020 19825 raft_consensus.cc:686] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 1 LEADER]: Becoming Leader. State: Replica: ccdabf4c519c4ea6847946859873ce70, State: Running, Role: LEADER
I20260806 02:54:48.053398 19825 consensus_queue.cc:237] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } }
I20260806 02:54:48.057018 19179 catalog_manager.cc:5495] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccdabf4c519c4ea6847946859873ce70 (127.24.130.130). New cstate: current_term: 1 leader_uuid: "ccdabf4c519c4ea6847946859873ce70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccdabf4c519c4ea6847946859873ce70" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 45281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20260806 02:54:48.060063 19616 tablet.cc:2369] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:48.116726 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:48.112819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.24.130.130:45281)
0806 02:54:48.112945 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:48.112949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:48.112966 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:54:48.113042 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:48.113044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:48.113103 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:48.113200 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:48.113498 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.114060 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.115011 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:48.115260 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:48.115386 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:48.115774 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:48.116258 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:48.116428 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:48.116526 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 02:54:48.116566 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:54:48.123150 19721 raft_consensus.cc:1261] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 1 FOLLOWER]: Refusing update from remote peer ccdabf4c519c4ea6847946859873ce70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:48.123152 19378 raft_consensus.cc:1261] T 26b75b66d6ae44e495f0ff0f93bd6ade P 637a1c67763f4d23b84640034b2b9102 [term 1 FOLLOWER]: Refusing update from remote peer ccdabf4c519c4ea6847946859873ce70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:48.123941 19825 consensus_queue.cc:1035] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "637a1c67763f4d23b84640034b2b9102" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:48.124194 19827 consensus_queue.cc:1035] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac4ba67589d48d3afddce4311890f2f" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:48.132639 19832 mvcc.cc:205] Tried to move back new op lower bound from 7315394101744304128 to 7315394101469749248. Current Snapshot: MvccSnapshot[applied={T|T < 7315394101744304128}]
I20260806 02:54:48.135684 19830 mvcc.cc:205] Tried to move back new op lower bound from 7315394101744304128 to 7315394101469749248. Current Snapshot: MvccSnapshot[applied={T|T < 7315394101744304128 or (T in {7315394101744304128})}]
I20260806 02:54:48.141382 19179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:48.141603 19179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:48.144316 19179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71fe51601e97405cb35ab6ded7f53b0c: Sending DeleteTablet for 3 replicas of tablet 26b75b66d6ae44e495f0ff0f93bd6ade
I20260806 02:54:48.145188 19352 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b75b66d6ae44e495f0ff0f93bd6ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:48 UTC) from {username='slave'} at 127.0.0.1:37394
I20260806 02:54:48.145304 19523 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b75b66d6ae44e495f0ff0f93bd6ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:48 UTC) from {username='slave'} at 127.0.0.1:37882
I20260806 02:54:48.145938 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 19271
I20260806 02:54:48.145947 19844 tablet_replica.cc:330] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: stopping tablet replica
I20260806 02:54:48.145992 19696 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b75b66d6ae44e495f0ff0f93bd6ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:48 UTC) from {username='slave'} at 127.0.0.1:57454
I20260806 02:54:48.146320 19844 raft_consensus.cc:2229] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:48.146797 19844 raft_consensus.cc:2258] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:48.148800 19845 tablet_replica.cc:330] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: stopping tablet replica
I20260806 02:54:48.149245 19845 raft_consensus.cc:2229] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:48.149686 19845 raft_consensus.cc:2258] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:48.157095 19844 ts_tablet_manager.cc:1905] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:48.159567 19844 ts_tablet_manager.cc:1918] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:48.159703 19844 log.cc:1196] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/26b75b66d6ae44e495f0ff0f93bd6ade
I20260806 02:54:48.160066 19844 ts_tablet_manager.cc:1939] T 26b75b66d6ae44e495f0ff0f93bd6ade P ccdabf4c519c4ea6847946859873ce70: Deleting consensus metadata
I20260806 02:54:48.161087 19845 ts_tablet_manager.cc:1905] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:48.161604 19154 catalog_manager.cc:4841] TS ccdabf4c519c4ea6847946859873ce70 (127.24.130.130:45281): tablet 26b75b66d6ae44e495f0ff0f93bd6ade (table test-table [id=ead1f64fbd60470abe713ff022f30e7d]) successfully deleted
W20260806 02:54:48.162503 19153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:42657 (error 108)
I20260806 02:54:48.162712 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 19446
I20260806 02:54:48.166045 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:48.165624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:42657 (local address 127.0.0.1:60322)
0806 02:54:48.165840 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:48.165929 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260806 02:54:48.166245 19153 catalog_manager.cc:4584] TS 637a1c67763f4d23b84640034b2b9102 (127.24.130.129:42657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26b75b66d6ae44e495f0ff0f93bd6ade: Network error: Client connection negotiation failed: client connection to 127.24.130.129:42657: connect: Connection refused (error 111)
I20260806 02:54:48.166935 19845 ts_tablet_manager.cc:1918] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:48.167142 19845 log.cc:1196] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/26b75b66d6ae44e495f0ff0f93bd6ade
I20260806 02:54:48.167631 19845 ts_tablet_manager.cc:1939] T 26b75b66d6ae44e495f0ff0f93bd6ade P 7ac4ba67589d48d3afddce4311890f2f: Deleting consensus metadata
I20260806 02:54:48.169626 19159 catalog_manager.cc:4841] TS 7ac4ba67589d48d3afddce4311890f2f (127.24.130.131:46119): tablet 26b75b66d6ae44e495f0ff0f93bd6ade (table test-table [id=ead1f64fbd60470abe713ff022f30e7d]) successfully deleted
I20260806 02:54:48.175115 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 19620
I20260806 02:54:48.187124 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 19129
2026-08-06T02:54:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1340 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:48 dist-test-slave-3kk6 krb5kdc[19851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:48 dist-test-slave-3kk6 krb5kdc[19851](info): set up 2 sockets
Aug 06 02:54:48 dist-test-slave-3kk6 krb5kdc[19851](info): commencing operation
krb5kdc: starting...
W20260806 02:54:49.417937 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:49 dist-test-slave-3kk6 krb5kdc[19851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:54:49.558352 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39033
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:49.680732 19869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:49.681056 19869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:49.681103 19869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:49.685292 19869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:49.685376 19869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:49.685396 19869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:49.685413 19869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:49.685429 19869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:49.691145 19869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:49.692411 19869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:49.693527 19869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:49.702771 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:49.702749 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:49.702747 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:49.702852 19869 server_base.cc:1034] running on GCE node
I20260806 02:54:49.703645 19869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:49.704913 19869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:49.706127 19869 hybrid_clock.cc:648] HybridClock initialized: now 1785984889706100 us; error 46 us; skew 500 ppm
Aug 06 02:54:49 dist-test-slave-3kk6 krb5kdc[19851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:49.709185 19869 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:54:49.710196 19869 webserver.cc:460] Webserver started at http://127.24.130.190:46531/ using document root <none> and password file <none>
I20260806 02:54:49.710870 19869 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:49.710968 19869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:49.711220 19869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:49.713436 19869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "874b310978d74cad9a0f4477d8b41ac9"
format_stamp: "Formatted at 2026-08-06 02:54:49 on dist-test-slave-3kk6"
server_key: "d2c97809638270d2f829e75155097c98"
server_key_iv: "453337f33672082b8ebe6ca9434cf5c8"
server_key_version: "encryptionkey@0"
I20260806 02:54:49.714061 19869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "874b310978d74cad9a0f4477d8b41ac9"
format_stamp: "Formatted at 2026-08-06 02:54:49 on dist-test-slave-3kk6"
server_key: "d2c97809638270d2f829e75155097c98"
server_key_iv: "453337f33672082b8ebe6ca9434cf5c8"
server_key_version: "encryptionkey@0"
I20260806 02:54:49.718178 19869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:54:49.721170 19891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:49.722616 19869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:54:49.722846 19869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "874b310978d74cad9a0f4477d8b41ac9"
format_stamp: "Formatted at 2026-08-06 02:54:49 on dist-test-slave-3kk6"
server_key: "d2c97809638270d2f829e75155097c98"
server_key_iv: "453337f33672082b8ebe6ca9434cf5c8"
server_key_version: "encryptionkey@0"
I20260806 02:54:49.722994 19869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:49.736579 19869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:49.740628 19869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:49.740899 19869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:49.749035 19869 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:39033
I20260806 02:54:49.749056 19961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:39033 every 8 connection(s)
I20260806 02:54:49.750147 19869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:49.753226 19963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:49.757545 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 19869
I20260806 02:54:49.757683 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:49.758019 25098 external_mini_cluster.cc:1428] Setting key f8e3522349a85af8d203cd7b7f2356b2
I20260806 02:54:49.759805 19963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9: Bootstrap starting.
I20260806 02:54:49.762573 19963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:49.763458 19963 log.cc:824] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:49.765821 19963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9: No bootstrap required, opened a new log
Aug 06 02:54:49 dist-test-slave-3kk6 krb5kdc[19851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:49.770561 19963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874b310978d74cad9a0f4477d8b41ac9" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39033 } }
I20260806 02:54:49.770911 19963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:49.770987 19963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874b310978d74cad9a0f4477d8b41ac9, State: Initialized, Role: FOLLOWER
I20260806 02:54:49.771526 19963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874b310978d74cad9a0f4477d8b41ac9" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39033 } }
I20260806 02:54:49.771664 19963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:49.771813 19963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:49.771951 19963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:49.773032 19963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874b310978d74cad9a0f4477d8b41ac9" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39033 } }
I20260806 02:54:49.773443 19963 leader_election.cc:302] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874b310978d74cad9a0f4477d8b41ac9; no voters:
I20260806 02:54:49.773846 19963 leader_election.cc:288] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:49.773936 19971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:49.774115 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:49.759688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.24.130.190:39033)
0806 02:54:49.760195 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:49.760213 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:49.760252 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:54:49.760951 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:49.760969 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:49.761307 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:49.761602 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:49.762751 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:49.763906 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:49.765060 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:49.765536 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:49.765659 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:49.768743 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:49.768749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:49.768764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:49.768794 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:49.770726 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:49.771288 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:49.771300 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:49.771305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:49.771433 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:49.771775 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:49.771781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:49.771783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:49.772302 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:49.772540 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:49.772808 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 02:54:49.773129 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":187,"threads_started":1}
I20260806 02:54:49.774175 19971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 874b310978d74cad9a0f4477d8b41ac9, State: Running, Role: LEADER
I20260806 02:54:49.774824 19971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874b310978d74cad9a0f4477d8b41ac9" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39033 } }
I20260806 02:54:49.775292 19963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:49.776717 19973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874b310978d74cad9a0f4477d8b41ac9. Latest consensus state: current_term: 1 leader_uuid: "874b310978d74cad9a0f4477d8b41ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874b310978d74cad9a0f4477d8b41ac9" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39033 } } }
I20260806 02:54:49.776856 19973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:49.777246 19972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874b310978d74cad9a0f4477d8b41ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874b310978d74cad9a0f4477d8b41ac9" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39033 } } }
I20260806 02:54:49.777359 19972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:49.778265 19980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:49.781016 19980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:49.786890 19980 catalog_manager.cc:1347] Generated new cluster ID: fa9a9dad71844dba809026f2bd70d042
I20260806 02:54:49.786978 19980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:49.800303 19980 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:49.801378 19980 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:49.810137 19980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9: Generated new TSK 0
I20260806 02:54:49.811093 19980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.130.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
Entry for principal kudu/127.24.130.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab.
I20260806 02:54:49.848174 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39033
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:49.969506 19998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:49.969847 19998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:49.969913 19998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:49.974224 19998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:49.974328 19998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:49.974393 19998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:49.980563 19998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.129.keytab
--principal=kudu/127.24.130.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:49.981842 19998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:49.983031 19998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:49.994036 20005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:49.993984 20006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:49.994007 20009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:49.994546 19998 server_base.cc:1034] running on GCE node
I20260806 02:54:49.995074 19998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:49.995860 19998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:49.997067 19998 hybrid_clock.cc:648] HybridClock initialized: now 1785984889997050 us; error 42 us; skew 500 ppm
Aug 06 02:54:49 dist-test-slave-3kk6 krb5kdc[19851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:50.000330 19998 init.cc:377] Logged in from keytab as kudu/127.24.130.129@KRBTEST.COM (short username kudu)
I20260806 02:54:50.001350 19998 webserver.cc:460] Webserver started at http://127.24.130.129:40631/ using document root <none> and password file <none>
I20260806 02:54:50.001945 19998 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:50.002010 19998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:50.002190 19998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:50.004511 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "a22c644731fb4ac99d2bc41d9d3c194a"
format_stamp: "Formatted at 2026-08-06 02:54:49 on dist-test-slave-3kk6"
server_key: "c0e594cc5393fe6f27944c3293150c4e"
server_key_iv: "ffe51662843f5ff77d7d6cf512353f51"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.005096 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "a22c644731fb4ac99d2bc41d9d3c194a"
format_stamp: "Formatted at 2026-08-06 02:54:49 on dist-test-slave-3kk6"
server_key: "c0e594cc5393fe6f27944c3293150c4e"
server_key_iv: "ffe51662843f5ff77d7d6cf512353f51"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.009235 19998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 02:54:50.011971 20017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.013322 19998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:54:50.013473 19998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "a22c644731fb4ac99d2bc41d9d3c194a"
format_stamp: "Formatted at 2026-08-06 02:54:49 on dist-test-slave-3kk6"
server_key: "c0e594cc5393fe6f27944c3293150c4e"
server_key_iv: "ffe51662843f5ff77d7d6cf512353f51"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.013597 19998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:50.037012 19998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:50.040563 19998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:50.040803 19998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:50.041502 19998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:50.042429 19998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:50.042480 19998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.042523 19998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:50.042577 19998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.052136 19998 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:45805
I20260806 02:54:50.052240 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:45805 every 8 connection(s)
I20260806 02:54:50.053136 19998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:50.057294 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 19998
I20260806 02:54:50.057454 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:50.057781 25098 external_mini_cluster.cc:1428] Setting key eacfbee679b9d4450dbe6618b93f2664
Aug 06 02:54:50 dist-test-slave-3kk6 krb5kdc[19851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984889, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.129@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:50.069171 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.055171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:43811 (local address 127.24.130.190:39033)
0806 02:54:50.055362 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.055367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.056282 (+ 915us) server_negotiation.cc:407] Connection header received
0806 02:54:50.057727 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.057731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.057795 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.057917 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:50.059509 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.060050 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.061215 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.061594 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.061718 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.064843 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:50.064848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:50.064853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:50.064898 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:50.066872 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.067402 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.067408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.067411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.067516 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.067871 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.067878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.067881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.068155 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:50.068271 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.068787 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.068967 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.24.130.130@KRBTEST.COM; defaulting to no policy
I20260806 02:54:50.070017 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.055616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39033 (local address 127.24.130.129:43811)
0806 02:54:50.056110 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.056158 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.057479 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.058097 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.058104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.058608 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:50.059312 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.059333 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.060186 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.060188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.061079 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.061088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.061783 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.061786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.061951 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.062712 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:50.062743 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:50.064628 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:50.067004 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:50.067009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:50.067022 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:50.067296 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:50.067616 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:50.067619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:50.067620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:50.067768 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:50.068265 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:50.068270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:50.068524 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.068693 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.069024 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260806 02:54:50.071604 20172 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39033
I20260806 02:54:50.071928 20172 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.130@KRBTEST.COM" created.
I20260806 02:54:50.072548 20172 heartbeater.cc:511] Master 127.24.130.190:39033 requested a full tablet report, sending...
I20260806 02:54:50.074443 19914 ts_manager.cc:194] Registered new tserver with Master: a22c644731fb4ac99d2bc41d9d3c194a (127.24.130.129:45805)
I20260806 02:54:50.075850 19914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.129@KRBTEST.COM'} at 127.24.130.129:43811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
Entry for principal kudu/127.24.130.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab.
I20260806 02:54:50.089884 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39033
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:50.213342 20181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:50.213738 20181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:50.213824 20181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:50.218209 20181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:50.218320 20181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:50.218421 20181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:50.224541 20181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.130.keytab
--principal=kudu/127.24.130.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:50.225819 20181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:50.227095 20181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:50.237521 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:50.237520 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:50.237591 20181 server_base.cc:1034] running on GCE node
W20260806 02:54:50.237532 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:50.238309 20181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:50.239107 20181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:50.240311 20181 hybrid_clock.cc:648] HybridClock initialized: now 1785984890240295 us; error 31 us; skew 500 ppm
Aug 06 02:54:50 dist-test-slave-3kk6 krb5kdc[19851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:50.243649 20181 init.cc:377] Logged in from keytab as kudu/127.24.130.130@KRBTEST.COM (short username kudu)
I20260806 02:54:50.244659 20181 webserver.cc:460] Webserver started at http://127.24.130.130:38571/ using document root <none> and password file <none>
I20260806 02:54:50.245275 20181 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:50.245339 20181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:50.245609 20181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:50.247879 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "d378b18969d14a02845d513d583c689b"
format_stamp: "Formatted at 2026-08-06 02:54:50 on dist-test-slave-3kk6"
server_key: "6c3442e164ae72c5ecc688e6a877d8f4"
server_key_iv: "577ca28c5f229b59d0b1dbdd2ece5daa"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.248490 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "d378b18969d14a02845d513d583c689b"
format_stamp: "Formatted at 2026-08-06 02:54:50 on dist-test-slave-3kk6"
server_key: "6c3442e164ae72c5ecc688e6a877d8f4"
server_key_iv: "577ca28c5f229b59d0b1dbdd2ece5daa"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.252465 20181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:54:50.255055 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.256366 20181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:54:50.256616 20181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "d378b18969d14a02845d513d583c689b"
format_stamp: "Formatted at 2026-08-06 02:54:50 on dist-test-slave-3kk6"
server_key: "6c3442e164ae72c5ecc688e6a877d8f4"
server_key_iv: "577ca28c5f229b59d0b1dbdd2ece5daa"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.256745 20181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:50.275558 20181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:50.279476 20181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:50.279714 20181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:50.280437 20181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:50.281340 20181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:50.281395 20181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.281440 20181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:50.281454 20181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.290943 20181 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:42223
I20260806 02:54:50.290968 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:42223 every 8 connection(s)
I20260806 02:54:50.292007 20181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:50.298189 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 20181
I20260806 02:54:50.298388 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:50.298731 25098 external_mini_cluster.cc:1428] Setting key 461e68cb4e8458efc6eca2cc825df2de
Aug 06 02:54:50 dist-test-slave-3kk6 krb5kdc[19851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.130@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:50.309383 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.294073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:43451 (local address 127.24.130.190:39033)
0806 02:54:50.294228 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.294232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.295256 (+ 1024us) server_negotiation.cc:407] Connection header received
0806 02:54:50.296670 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.296678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.296734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.296823 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:50.298333 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.299182 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.300817 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.301058 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.301143 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.304677 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:50.304685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:50.304690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:50.304729 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:50.306802 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.307448 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.307455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.307460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.307575 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.307972 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.307978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.307983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.308302 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:50.308425 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.309000 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.309169 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:54:50.310341 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.294492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39033 (local address 127.24.130.130:43451)
0806 02:54:50.295064 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.295114 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.296400 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.297003 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.297010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.297518 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:50.298131 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.298147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.299364 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.299368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.300675 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.300690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.301234 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.301241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.301508 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.302244 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:50.302272 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:50.304456 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:50.306955 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:50.306962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:50.306980 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:50.307316 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:50.307685 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:50.307687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:50.307689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:50.307864 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:50.308418 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:50.308424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:50.308715 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.308891 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.309271 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":207,"threads_started":1}
WARNING: no policy specified for kudu/127.24.130.131@KRBTEST.COM; defaulting to no policy
I20260806 02:54:50.311786 20358 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39033
I20260806 02:54:50.312062 20358 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:50.312690 20358 heartbeater.cc:511] Master 127.24.130.190:39033 requested a full tablet report, sending...
I20260806 02:54:50.313952 19914 ts_manager.cc:194] Registered new tserver with Master: d378b18969d14a02845d513d583c689b (127.24.130.130:42223)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.131@KRBTEST.COM" created.
I20260806 02:54:50.314797 19914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.130@KRBTEST.COM'} at 127.24.130.130:43451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
Entry for principal kudu/127.24.130.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab.
I20260806 02:54:50.333564 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39033
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:50.453125 20363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:50.453469 20363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:50.453552 20363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:50.457664 20363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:50.457782 20363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:50.457886 20363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:50.463606 20363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.131.keytab
--principal=kudu/127.24.130.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:50.464835 20363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:50.465981 20363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:50.476125 20374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:50.476125 20372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:50.476253 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:50.476662 20363 server_base.cc:1034] running on GCE node
I20260806 02:54:50.477128 20363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:50.477815 20363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:50.479009 20363 hybrid_clock.cc:648] HybridClock initialized: now 1785984890478990 us; error 33 us; skew 500 ppm
Aug 06 02:54:50 dist-test-slave-3kk6 krb5kdc[19851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:50.481961 20363 init.cc:377] Logged in from keytab as kudu/127.24.130.131@KRBTEST.COM (short username kudu)
I20260806 02:54:50.482956 20363 webserver.cc:460] Webserver started at http://127.24.130.131:38913/ using document root <none> and password file <none>
I20260806 02:54:50.483556 20363 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:50.483652 20363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:50.483907 20363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:50.486117 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "e0678684cae24b49ac211ddd71d32de4"
format_stamp: "Formatted at 2026-08-06 02:54:50 on dist-test-slave-3kk6"
server_key: "33921184ee80117585feda48efd2d2fd"
server_key_iv: "1c455bee2d034f49b9320177415f8d77"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.486804 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "e0678684cae24b49ac211ddd71d32de4"
format_stamp: "Formatted at 2026-08-06 02:54:50 on dist-test-slave-3kk6"
server_key: "33921184ee80117585feda48efd2d2fd"
server_key_iv: "1c455bee2d034f49b9320177415f8d77"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.490823 20363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:54:50.493327 20387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.494720 20363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:54:50.494863 20363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "e0678684cae24b49ac211ddd71d32de4"
format_stamp: "Formatted at 2026-08-06 02:54:50 on dist-test-slave-3kk6"
server_key: "33921184ee80117585feda48efd2d2fd"
server_key_iv: "1c455bee2d034f49b9320177415f8d77"
server_key_version: "encryptionkey@0"
I20260806 02:54:50.494992 20363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:50.516280 20363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:50.519946 20363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:50.520190 20363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:50.520867 20363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:50.521947 20363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:50.522022 20363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.522094 20363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:50.522146 20363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:50.532860 20363 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:42413
I20260806 02:54:50.532881 20543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:42413 every 8 connection(s)
I20260806 02:54:50.533900 20363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:50.542461 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 20363
I20260806 02:54:50.542639 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:50.543038 25098 external_mini_cluster.cc:1428] Setting key 19b83baec4aa3b5fafd4f062c5f8f8d7
Aug 06 02:54:50 dist-test-slave-3kk6 krb5kdc[19851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984890, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.131@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:54:50.549702 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.536051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:34893 (local address 127.24.130.190:39033)
0806 02:54:50.536219 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.536224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.537197 (+ 973us) server_negotiation.cc:407] Connection header received
0806 02:54:50.538795 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.538798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.538854 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.538929 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:50.540347 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.540915 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.542162 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.542510 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.542637 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.545815 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:50.545819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:50.545822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:50.545858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:50.547373 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.547970 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.547975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.547977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.548070 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.548457 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.548461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.548462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.548675 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:50.548790 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.549372 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.549526 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:54:50.550633 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.536492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:39033 (local address 127.24.130.131:34893)
0806 02:54:50.537017 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.537070 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.538378 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.539171 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.539179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.539610 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:50.540157 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.540172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.541042 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.541044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.542017 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.542027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.542768 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.542774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.542990 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.543759 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:50.543835 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:50.545629 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:50.547522 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:50.547529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:50.547548 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:50.547850 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:50.548180 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:50.548183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:50.548185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:50.548352 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:50.548791 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:50.548801 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:50.549111 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.549328 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.549714 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260806 02:54:50.552028 20544 heartbeater.cc:347] Connected to a master server at 127.24.130.190:39033
I20260806 02:54:50.552309 20544 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:50.552903 20544 heartbeater.cc:511] Master 127.24.130.190:39033 requested a full tablet report, sending...
I20260806 02:54:50.554060 19914 ts_manager.cc:194] Registered new tserver with Master: e0678684cae24b49ac211ddd71d32de4 (127.24.130.131:42413)
I20260806 02:54:50.554757 19914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.130.131@KRBTEST.COM'} at 127.24.130.131:34893
I20260806 02:54:50.557529 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:50.567553 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.559718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.24.130.190:39033)
0806 02:54:50.559878 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.559882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.560034 (+ 152us) server_negotiation.cc:407] Connection header received
0806 02:54:50.560165 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.560167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.560209 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.560277 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:50.561052 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.561628 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.562450 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.562741 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.562832 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.564355 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:50.564357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:50.564359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:50.564378 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:50.565704 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.566110 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.566112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.566113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.566192 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:50.566442 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:50.566444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:50.566445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:50.566843 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:50.566972 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.567196 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.567312 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:54:50.571230 19914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:50.574373 19914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:50.590719 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.584043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42223 (local address 127.0.0.1:49726)
0806 02:54:50.584675 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.584732 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.584894 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.586114 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.586119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.586140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:50.586220 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.586225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.587845 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.587848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.588976 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.588982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.590149 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.590153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.590350 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.590370 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.590426 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.590492 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":140,"thread_start_us":128,"threads_started":1}
I20260806 02:54:50.590739 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.584117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:45805 (local address 127.0.0.1:48310)
0806 02:54:50.584895 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.584923 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.585024 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.585791 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.585798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.585823 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:50.586020 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.586033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.587698 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.587701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.588967 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.588980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.590149 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.590153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.590291 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.590342 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.590414 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.590502 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":162,"thread_start_us":112,"threads_started":1}
I20260806 02:54:50.591012 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.584174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.24.130.130:42223)
0806 02:54:50.585626 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.585644 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.585682 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:54:50.585767 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.585772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.585943 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.586363 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:50.586555 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.587683 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.589110 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.589990 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.590244 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.590390 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.590479 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.590869 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":109,"threads_started":1}
I20260806 02:54:50.591250 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.584962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.24.130.129:45805)
0806 02:54:50.585246 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.585255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.585283 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:54:50.585361 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.585367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.585601 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.586197 (+ 596us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:50.586359 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.587560 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.589110 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.589993 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.590793 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.590943 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.590999 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.591108 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260806 02:54:50.591342 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.584965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.24.130.131:42413)
0806 02:54:50.585512 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.585526 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.585553 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:54:50.585632 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.585637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.585820 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.586277 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:50.586389 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.587232 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.589233 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.589852 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.590009 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.590161 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.590596 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.590752 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":160,"threads_started":1}
I20260806 02:54:50.592394 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.584403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:42413 (local address 127.0.0.1:51796)
0806 02:54:50.585141 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.585156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.585235 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.585995 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.585998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.586016 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:50.586132 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.586139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.587372 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.587378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.589124 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.589129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.589993 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.589996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.590118 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.590161 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.590281 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.590439 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":260,"thread_start_us":108,"threads_started":1}
I20260806 02:54:50.594589 20088 tablet_service.cc:1467] Processing CreateTablet for tablet 97f05e20dd06482ca416fe36d46d96a9 (DEFAULT_TABLE table=test-table [id=477950fb00ad4ab98bfe70367fac41d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:50.594589 20274 tablet_service.cc:1467] Processing CreateTablet for tablet 97f05e20dd06482ca416fe36d46d96a9 (DEFAULT_TABLE table=test-table [id=477950fb00ad4ab98bfe70367fac41d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:50.596109 20088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f05e20dd06482ca416fe36d46d96a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:50.596109 20274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f05e20dd06482ca416fe36d46d96a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:50.597671 20447 tablet_service.cc:1467] Processing CreateTablet for tablet 97f05e20dd06482ca416fe36d46d96a9 (DEFAULT_TABLE table=test-table [id=477950fb00ad4ab98bfe70367fac41d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:50.598922 20447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f05e20dd06482ca416fe36d46d96a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:50.604326 20566 tablet_bootstrap.cc:492] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: Bootstrap starting.
I20260806 02:54:50.605705 20568 tablet_bootstrap.cc:492] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Bootstrap starting.
I20260806 02:54:50.607071 20569 tablet_bootstrap.cc:492] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Bootstrap starting.
I20260806 02:54:50.607187 20566 tablet_bootstrap.cc:654] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:50.608209 20568 tablet_bootstrap.cc:654] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:50.608181 20566 log.cc:824] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:50.609388 20569 tablet_bootstrap.cc:654] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:50.609571 20568 log.cc:824] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:50.610579 20566 tablet_bootstrap.cc:492] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: No bootstrap required, opened a new log
I20260806 02:54:50.610741 20569 log.cc:824] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:50.610863 20566 ts_tablet_manager.cc:1397] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:54:50.612037 20568 tablet_bootstrap.cc:492] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: No bootstrap required, opened a new log
I20260806 02:54:50.612246 20568 ts_tablet_manager.cc:1397] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260806 02:54:50.612833 20569 tablet_bootstrap.cc:492] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: No bootstrap required, opened a new log
I20260806 02:54:50.613020 20569 ts_tablet_manager.cc:1397] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:54:50.615548 20566 raft_consensus.cc:348] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.615839 20566 raft_consensus.cc:374] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:50.615911 20566 raft_consensus.cc:729] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22c644731fb4ac99d2bc41d9d3c194a, State: Initialized, Role: FOLLOWER
I20260806 02:54:50.616398 20566 consensus_queue.cc:260] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.616279 20568 raft_consensus.cc:348] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.616554 20568 raft_consensus.cc:374] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:50.616632 20568 raft_consensus.cc:729] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d378b18969d14a02845d513d583c689b, State: Initialized, Role: FOLLOWER
I20260806 02:54:50.616936 20569 raft_consensus.cc:348] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.617201 20569 raft_consensus.cc:374] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:50.617264 20569 raft_consensus.cc:729] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0678684cae24b49ac211ddd71d32de4, State: Initialized, Role: FOLLOWER
I20260806 02:54:50.617257 20568 consensus_queue.cc:260] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.617553 20566 ts_tablet_manager.cc:1428] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:54:50.617690 20569 consensus_queue.cc:260] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.617974 20172 heartbeater.cc:503] Master 127.24.130.190:39033 was elected leader, sending a full tablet report...
I20260806 02:54:50.618072 20358 heartbeater.cc:503] Master 127.24.130.190:39033 was elected leader, sending a full tablet report...
I20260806 02:54:50.618368 20568 ts_tablet_manager.cc:1428] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260806 02:54:50.619359 20544 heartbeater.cc:503] Master 127.24.130.190:39033 was elected leader, sending a full tablet report...
I20260806 02:54:50.619449 20569 ts_tablet_manager.cc:1428] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260806 02:54:50.781380 20581 raft_consensus.cc:482] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:50.781683 20581 raft_consensus.cc:504] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.783020 20581 leader_election.cc:288] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d378b18969d14a02845d513d583c689b (127.24.130.130:42223), a22c644731fb4ac99d2bc41d9d3c194a (127.24.130.129:45805)
W20260806 02:54:50.785008 20545 tablet.cc:2369] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:50.787109 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.783396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:33045 (local address 127.24.130.130:42223)
0806 02:54:50.783541 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.783545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.783564 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:54:50.783729 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.783733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.783802 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.783893 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:50.784221 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.784882 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.786088 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.786490 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.786626 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.786708 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.786915 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.786994 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:54:50.787110 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.783261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42223 (local address 127.24.130.131:33045)
0806 02:54:50.783471 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.783490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.783593 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.783919 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.783922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.783963 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:50.784095 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.784104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.785013 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.785019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.785946 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.785956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.786620 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.786624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.786763 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.786792 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.786886 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.786967 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 02:54:50.787226 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.783508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:47393 (local address 127.24.130.129:45805)
0806 02:54:50.783654 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.783660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.783878 (+ 218us) server_negotiation.cc:407] Connection header received
0806 02:54:50.784021 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.784024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.784081 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.784173 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:50.784492 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.785013 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.786179 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.786610 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.786784 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.786835 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.787025 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.787099 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:54:50.787271 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.783320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:45805 (local address 127.24.130.131:47393)
0806 02:54:50.783753 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.783768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:50.783905 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:50.784234 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:50.784237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:50.784256 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:50.784379 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.784385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.785133 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.785136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.786038 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:50.786047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.786783 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.786786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:50.786893 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.786913 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:54:50.787007 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:54:50.787080 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":143,"threads_started":1}
I20260806 02:54:50.787981 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f05e20dd06482ca416fe36d46d96a9" candidate_uuid: "e0678684cae24b49ac211ddd71d32de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d378b18969d14a02845d513d583c689b" is_pre_election: true
I20260806 02:54:50.788015 20115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f05e20dd06482ca416fe36d46d96a9" candidate_uuid: "e0678684cae24b49ac211ddd71d32de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" is_pre_election: true
I20260806 02:54:50.788363 20297 raft_consensus.cc:2454] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0678684cae24b49ac211ddd71d32de4 in term 0.
I20260806 02:54:50.788363 20115 raft_consensus.cc:2454] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0678684cae24b49ac211ddd71d32de4 in term 0.
I20260806 02:54:50.788902 20394 leader_election.cc:302] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a22c644731fb4ac99d2bc41d9d3c194a, d378b18969d14a02845d513d583c689b, e0678684cae24b49ac211ddd71d32de4; no voters:
I20260806 02:54:50.789207 20581 raft_consensus.cc:2781] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:50.789289 20581 raft_consensus.cc:482] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:50.789321 20581 raft_consensus.cc:3037] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:50.790475 20581 raft_consensus.cc:504] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.790911 20581 leader_election.cc:288] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [CANDIDATE]: Term 1 election: Requested vote from peers d378b18969d14a02845d513d583c689b (127.24.130.130:42223), a22c644731fb4ac99d2bc41d9d3c194a (127.24.130.129:45805)
I20260806 02:54:50.791321 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f05e20dd06482ca416fe36d46d96a9" candidate_uuid: "e0678684cae24b49ac211ddd71d32de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d378b18969d14a02845d513d583c689b"
I20260806 02:54:50.791489 20297 raft_consensus.cc:3037] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:50.791507 20115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f05e20dd06482ca416fe36d46d96a9" candidate_uuid: "e0678684cae24b49ac211ddd71d32de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22c644731fb4ac99d2bc41d9d3c194a"
I20260806 02:54:50.791652 20115 raft_consensus.cc:3037] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:50.792670 20297 raft_consensus.cc:2454] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0678684cae24b49ac211ddd71d32de4 in term 1.
I20260806 02:54:50.792738 20115 raft_consensus.cc:2454] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0678684cae24b49ac211ddd71d32de4 in term 1.
I20260806 02:54:50.793043 20394 leader_election.cc:302] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d378b18969d14a02845d513d583c689b, e0678684cae24b49ac211ddd71d32de4; no voters:
I20260806 02:54:50.793247 20581 raft_consensus.cc:2781] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 02:54:50.793443 20359 tablet.cc:2369] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:50.793519 20581 raft_consensus.cc:686] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 1 LEADER]: Becoming Leader. State: Replica: e0678684cae24b49ac211ddd71d32de4, State: Running, Role: LEADER
I20260806 02:54:50.793895 20581 consensus_queue.cc:237] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } }
I20260806 02:54:50.797428 19910 catalog_manager.cc:5495] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0678684cae24b49ac211ddd71d32de4 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "e0678684cae24b49ac211ddd71d32de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0678684cae24b49ac211ddd71d32de4" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 42413 } health_report { overall_health: HEALTHY } } }
W20260806 02:54:50.804589 20174 tablet.cc:2369] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:50.837149 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.833448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.24.130.131:42413)
0806 02:54:50.833592 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:50.833595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:50.833611 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:54:50.833692 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:50.833694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:50.833753 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:50.833866 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:50.834211 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.834808 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.835676 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:50.835898 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:50.835982 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:50.836296 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:50.836764 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:50.836862 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:50.836943 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 02:54:50.837002 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:54:50.843663 20115 raft_consensus.cc:1261] T 97f05e20dd06482ca416fe36d46d96a9 P a22c644731fb4ac99d2bc41d9d3c194a [term 1 FOLLOWER]: Refusing update from remote peer e0678684cae24b49ac211ddd71d32de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:50.843663 20297 raft_consensus.cc:1261] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 1 FOLLOWER]: Refusing update from remote peer e0678684cae24b49ac211ddd71d32de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:50.844498 20583 consensus_queue.cc:1035] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22c644731fb4ac99d2bc41d9d3c194a" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:50.844707 20581 consensus_queue.cc:1035] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d378b18969d14a02845d513d583c689b" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:50.852900 20587 mvcc.cc:205] Tried to move back new op lower bound from 7315394112887308288 to 7315394112694681600. Current Snapshot: MvccSnapshot[applied={T|T < 7315394112887308288}]
I20260806 02:54:50.854525 20588 mvcc.cc:205] Tried to move back new op lower bound from 7315394112887308288 to 7315394112694681600. Current Snapshot: MvccSnapshot[applied={T|T < 7315394112887308288}]
I20260806 02:54:50.863029 19912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:50.863216 19912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:50.865885 19912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874b310978d74cad9a0f4477d8b41ac9: Sending DeleteTablet for 3 replicas of tablet 97f05e20dd06482ca416fe36d46d96a9
I20260806 02:54:50.866854 20274 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f05e20dd06482ca416fe36d46d96a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:50 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:49726
I20260806 02:54:50.866935 20088 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f05e20dd06482ca416fe36d46d96a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:50 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:48310
I20260806 02:54:50.867487 20447 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f05e20dd06482ca416fe36d46d96a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:50 UTC) from {username='kudu', principal='kudu/127.24.130.190@KRBTEST.COM'} at 127.0.0.1:51796
I20260806 02:54:50.867517 20600 tablet_replica.cc:330] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: stopping tablet replica
I20260806 02:54:50.867918 20600 raft_consensus.cc:2229] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:50.868069 20601 tablet_replica.cc:330] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: stopping tablet replica
I20260806 02:54:50.868306 20600 raft_consensus.cc:2258] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:50.868647 20601 raft_consensus.cc:2229] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:50.869092 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 19998
I20260806 02:54:50.869261 20601 raft_consensus.cc:2258] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:50.870334 20600 ts_tablet_manager.cc:1905] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:50.873473 20600 ts_tablet_manager.cc:1918] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:50.873591 20600 log.cc:1196] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/97f05e20dd06482ca416fe36d46d96a9
I20260806 02:54:50.873940 20600 ts_tablet_manager.cc:1939] T 97f05e20dd06482ca416fe36d46d96a9 P d378b18969d14a02845d513d583c689b: Deleting consensus metadata
I20260806 02:54:50.876293 20601 ts_tablet_manager.cc:1905] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:50.879328 20601 ts_tablet_manager.cc:1918] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:50.879451 20601 log.cc:1196] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/97f05e20dd06482ca416fe36d46d96a9
I20260806 02:54:50.879796 20601 ts_tablet_manager.cc:1939] T 97f05e20dd06482ca416fe36d46d96a9 P e0678684cae24b49ac211ddd71d32de4: Deleting consensus metadata
I20260806 02:54:50.881183 19893 catalog_manager.cc:4841] TS e0678684cae24b49ac211ddd71d32de4 (127.24.130.131:42413): tablet 97f05e20dd06482ca416fe36d46d96a9 (table test-table [id=477950fb00ad4ab98bfe70367fac41d2]) successfully deleted
I20260806 02:54:50.883204 19894 catalog_manager.cc:4841] TS d378b18969d14a02845d513d583c689b (127.24.130.130:42223): tablet 97f05e20dd06482ca416fe36d46d96a9 (table test-table [id=477950fb00ad4ab98bfe70367fac41d2]) successfully deleted
W20260806 02:54:50.885991 19896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:45805 (error 108)
I20260806 02:54:50.886281 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 20181
I20260806 02:54:50.908305 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:50.907811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:45805 (local address 127.0.0.1:48318)
0806 02:54:50.908079 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:50.908184 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260806 02:54:50.908615 19896 catalog_manager.cc:4584] TS a22c644731fb4ac99d2bc41d9d3c194a (127.24.130.129:45805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97f05e20dd06482ca416fe36d46d96a9: Network error: Client connection negotiation failed: client connection to 127.24.130.129:45805: connect: Connection refused (error 111)
I20260806 02:54:50.914901 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 20363
I20260806 02:54:50.930313 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 19869
2026-08-06T02:54:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2734 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T02:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:50Z Disabled control of system clock
I20260806 02:54:50.965632 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:37331
--webserver_interface=127.24.130.190
--webserver_port=0
--builtin_ntp_servers=127.24.130.148:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:54:51.084760 20608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:51.085114 20608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:51.085152 20608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:51.089298 20608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:51.089380 20608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:51.089396 20608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:51.089416 20608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:51.089432 20608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:51.094404 20608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:37331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:51.095783 20608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:51.097329 20608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:51.107052 20618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.107045 20621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.107048 20619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:51.107326 20608 server_base.cc:1034] running on GCE node
I20260806 02:54:51.107939 20608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:51.109479 20608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:51.110690 20608 hybrid_clock.cc:648] HybridClock initialized: now 1785984891110622 us; error 49 us; skew 500 ppm
I20260806 02:54:51.116931 20608 webserver.cc:460] Webserver started at http://127.24.130.190:40539/ using document root <none> and password file <none>
I20260806 02:54:51.117628 20608 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:51.117704 20608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:51.117972 20608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:51.120375 20608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "f7bacaa2c26ded8d74e402b01cec209f"
server_key_iv: "96228ae5145f8fc7fa619eaffb1700f3"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.121042 20608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "f7bacaa2c26ded8d74e402b01cec209f"
server_key_iv: "96228ae5145f8fc7fa619eaffb1700f3"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.125327 20608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:54:51.128089 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.129484 20608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:54:51.129655 20608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "f7bacaa2c26ded8d74e402b01cec209f"
server_key_iv: "96228ae5145f8fc7fa619eaffb1700f3"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.129801 20608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:51.150168 20608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:51.151109 20608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:51.151343 20608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:51.158842 20608 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:37331
I20260806 02:54:51.158849 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:37331 every 8 connection(s)
I20260806 02:54:51.159909 20608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:51.163578 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 20608
I20260806 02:54:51.163729 20699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:51.163740 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:51.164089 25098 external_mini_cluster.cc:1428] Setting key dd90e088e847c7a75ece289a36c60ab5
I20260806 02:54:51.171339 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708: Bootstrap starting.
I20260806 02:54:51.173840 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.165575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.24.130.190:37331)
0806 02:54:51.166238 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.166254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.166291 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:54:51.167088 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.167107 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.167524 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.167869 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:51.168313 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.169302 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.170234 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.170802 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.170991 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.172008 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:51.172014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:51.172028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:51.172444 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:51.172501 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:51.172520 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:51.172635 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.172799 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.173088 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
I20260806 02:54:51.173877 20699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:51.174837 20699 log.cc:824] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:51.177099 20699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708: No bootstrap required, opened a new log
I20260806 02:54:51.180873 20699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37331 } }
I20260806 02:54:51.181164 20699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:51.181214 20699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ebfd3e8dc5a49e1afb7b65f23d43708, State: Initialized, Role: FOLLOWER
I20260806 02:54:51.181723 20699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37331 } }
I20260806 02:54:51.181933 20699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:51.182022 20699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:51.182183 20699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:51.183442 20699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37331 } }
I20260806 02:54:51.183843 20699 leader_election.cc:302] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ebfd3e8dc5a49e1afb7b65f23d43708; no voters:
I20260806 02:54:51.184283 20699 leader_election.cc:288] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:51.184371 20705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:51.184757 20705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [term 1 LEADER]: Becoming Leader. State: Replica: 1ebfd3e8dc5a49e1afb7b65f23d43708, State: Running, Role: LEADER
I20260806 02:54:51.185101 20705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37331 } }
I20260806 02:54:51.185806 20699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:51.187077 20708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ebfd3e8dc5a49e1afb7b65f23d43708. Latest consensus state: current_term: 1 leader_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37331 } } }
I20260806 02:54:51.187076 20706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfd3e8dc5a49e1afb7b65f23d43708" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37331 } } }
I20260806 02:54:51.187247 20708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:51.187247 20706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:51.187769 20715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:51.191172 20715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:51.197387 20715 catalog_manager.cc:1347] Generated new cluster ID: d14e4eed49e8439f97949aede08b337a
I20260806 02:54:51.197482 20715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:51.209846 20715 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:51.210935 20715 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:51.216905 20715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708: Generated new TSK 0
I20260806 02:54:51.217654 20715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:51.225167 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.129:0
--local_ip_for_outbound_sockets=127.24.130.129
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:37331
--builtin_ntp_servers=127.24.130.148:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:54:51.343302 20725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:51.343665 20725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:51.343745 20725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:51.347982 20725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:51.348097 20725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:51.348212 20725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.129
I20260806 02:54:51.353322 20725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.130.129
--webserver_port=0
--tserver_master_addrs=127.24.130.190:37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.129
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:51.354612 20725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:51.356205 20725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:54:51.366539 20725 server_base.cc:1034] running on GCE node
W20260806 02:54:51.366511 20734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.366562 20735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.366511 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:51.367347 20725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:51.368075 20725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:51.369261 20725 hybrid_clock.cc:648] HybridClock initialized: now 1785984891369231 us; error 47 us; skew 500 ppm
I20260806 02:54:51.371466 20725 webserver.cc:460] Webserver started at http://127.24.130.129:33561/ using document root <none> and password file <none>
I20260806 02:54:51.372073 20725 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:51.372140 20725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:51.372323 20725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:51.374536 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data/instance:
uuid: "06e3edf89b4041f894060e72a0418385"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "f46a904dd19b80d58babe8e643d49a91"
server_key_iv: "8a2d25ba9108b4f59da177b2a97a6767"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.375221 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance:
uuid: "06e3edf89b4041f894060e72a0418385"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "f46a904dd19b80d58babe8e643d49a91"
server_key_iv: "8a2d25ba9108b4f59da177b2a97a6767"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.379220 20725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:54:51.381734 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.383059 20725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:54:51.383163 20725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal
uuid: "06e3edf89b4041f894060e72a0418385"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "f46a904dd19b80d58babe8e643d49a91"
server_key_iv: "8a2d25ba9108b4f59da177b2a97a6767"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.383258 20725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:51.406064 20725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:51.406903 20725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:51.407081 20725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:51.407668 20725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:51.408658 20725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:51.408708 20725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.408751 20725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:51.408767 20725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.417878 20725 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.129:46611
I20260806 02:54:51.417953 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.129:46611 every 8 connection(s)
I20260806 02:54:51.419001 20725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:51.423704 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 20725
I20260806 02:54:51.423821 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:51.424070 25098 external_mini_cluster.cc:1428] Setting key de40ba67fbb1aaffa181c2cc69feb0bb
I20260806 02:54:51.426255 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.130:0
--local_ip_for_outbound_sockets=127.24.130.130
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:37331
--builtin_ntp_servers=127.24.130.148:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:54:51.430226 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.421013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:47611 (local address 127.24.130.190:37331)
0806 02:54:51.421206 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.421212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.422303 (+ 1091us) server_negotiation.cc:407] Connection header received
0806 02:54:51.424993 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.424996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.425049 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.425140 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:51.426135 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.426820 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.428051 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.428291 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.428412 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.429702 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:51.429705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:51.429707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:51.429721 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:51.429741 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:51.429749 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:51.429829 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.429996 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.430072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:54:51.431231 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.421428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:37331 (local address 127.24.130.129:47611)
0806 02:54:51.422124 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:51.422168 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:51.424766 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:51.425327 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:51.425334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:51.425441 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:51.425976 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.425990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.426978 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.426981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.427903 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.427914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.428474 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.428478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.428656 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.429385 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:51.429387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:51.429445 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:51.429519 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:51.429833 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:51.429846 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:51.429853 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:54:51.429996 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:54:51.430312 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20260806 02:54:51.432626 20903 heartbeater.cc:347] Connected to a master server at 127.24.130.190:37331
I20260806 02:54:51.432922 20903 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:51.433727 20903 heartbeater.cc:511] Master 127.24.130.190:37331 requested a full tablet report, sending...
I20260806 02:54:51.435693 20652 ts_manager.cc:194] Registered new tserver with Master: 06e3edf89b4041f894060e72a0418385 (127.24.130.129:46611)
I20260806 02:54:51.437511 20652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.129:47611
W20260806 02:54:51.549793 20906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:51.550124 20906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:51.550165 20906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:51.554416 20906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:51.554489 20906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:51.554551 20906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.130
I20260806 02:54:51.559479 20906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.130.130
--webserver_port=0
--tserver_master_addrs=127.24.130.190:37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.130
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:51.560613 20906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:51.562067 20906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:51.572305 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:51.572465 20906 server_base.cc:1034] running on GCE node
W20260806 02:54:51.572305 20912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.572299 20911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:51.573132 20906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:51.573846 20906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:51.575044 20906 hybrid_clock.cc:648] HybridClock initialized: now 1785984891575017 us; error 44 us; skew 500 ppm
I20260806 02:54:51.577051 20906 webserver.cc:460] Webserver started at http://127.24.130.130:34173/ using document root <none> and password file <none>
I20260806 02:54:51.577694 20906 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:51.577800 20906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:51.578060 20906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:51.580379 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data/instance:
uuid: "f5946b20e5ef446c96086daf748e9d9a"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "6de3e03fcb1dd4b7d4deaf26dcc9b2c8"
server_key_iv: "6a6f93946d699e01aac1774454fbc77a"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.581033 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance:
uuid: "f5946b20e5ef446c96086daf748e9d9a"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "6de3e03fcb1dd4b7d4deaf26dcc9b2c8"
server_key_iv: "6a6f93946d699e01aac1774454fbc77a"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.585150 20906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 02:54:51.588297 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.589576 20906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:54:51.589726 20906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal
uuid: "f5946b20e5ef446c96086daf748e9d9a"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "6de3e03fcb1dd4b7d4deaf26dcc9b2c8"
server_key_iv: "6a6f93946d699e01aac1774454fbc77a"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.589856 20906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:51.602280 20906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:51.603118 20906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:51.603341 20906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:51.603968 20906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:51.605020 20906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:51.605105 20906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.605180 20906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:51.605225 20906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.614302 20906 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.130:42771
I20260806 02:54:51.614321 21078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.130:42771 every 8 connection(s)
I20260806 02:54:51.615419 20906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:51.625005 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 20906
I20260806 02:54:51.625167 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:51.625242 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.617402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.130:49463 (local address 127.24.130.190:37331)
0806 02:54:51.617587 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.617593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.618715 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 02:54:51.620183 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.620185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.620237 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.620317 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:51.621268 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.621871 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.623052 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.623269 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.623352 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.624664 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:51.624669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:51.624671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:51.624689 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:51.624717 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:51.624731 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:51.624816 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.624986 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.625062 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:54:51.625543 25098 external_mini_cluster.cc:1428] Setting key 47c9ca15e137fe9dfef4850cf6e398e2
I20260806 02:54:51.626302 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.617836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:37331 (local address 127.24.130.130:49463)
0806 02:54:51.618529 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:51.618573 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:51.619977 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:51.620519 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:51.620525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:51.620611 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:51.621120 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.621133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.622024 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.622026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.622939 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.622948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.623440 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.623442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.623588 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.624362 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:51.624364 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:51.624414 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:51.624483 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:51.624828 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:51.624835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:51.624840 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:54:51.624943 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:54:51.625240 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260806 02:54:51.627502 21080 heartbeater.cc:347] Connected to a master server at 127.24.130.190:37331
I20260806 02:54:51.627761 21080 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:51.627864 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.130.131:0
--local_ip_for_outbound_sockets=127.24.130.131
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:37331
--builtin_ntp_servers=127.24.130.148:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:54:51.628430 21080 heartbeater.cc:511] Master 127.24.130.190:37331 requested a full tablet report, sending...
I20260806 02:54:51.629576 20652 ts_manager.cc:194] Registered new tserver with Master: f5946b20e5ef446c96086daf748e9d9a (127.24.130.130:42771)
I20260806 02:54:51.630170 20652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.130:49463
W20260806 02:54:51.749843 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:51.750200 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:51.750278 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:51.754487 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:51.754598 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:51.754719 21087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.130.131
I20260806 02:54:51.759810 21087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.130.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.130.131
--webserver_port=0
--tserver_master_addrs=127.24.130.190:37331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.130.131
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:51.761049 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:51.762519 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:51.772787 21098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.772782 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:51.772781 21096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:51.773039 21087 server_base.cc:1034] running on GCE node
I20260806 02:54:51.773555 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:51.774250 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:51.775419 21087 hybrid_clock.cc:648] HybridClock initialized: now 1785984891775405 us; error 31 us; skew 500 ppm
I20260806 02:54:51.777468 21087 webserver.cc:460] Webserver started at http://127.24.130.131:45999/ using document root <none> and password file <none>
I20260806 02:54:51.778079 21087 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:51.778147 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:51.778434 21087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:51.780642 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data/instance:
uuid: "da3c14bccdb243c5a3509967f48b5fb2"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "615d14abde98aaf42071f52233933dbc"
server_key_iv: "ede800de39009aafa846a53f7642d6a9"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.781257 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance:
uuid: "da3c14bccdb243c5a3509967f48b5fb2"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "615d14abde98aaf42071f52233933dbc"
server_key_iv: "ede800de39009aafa846a53f7642d6a9"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.785256 21087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:54:51.788172 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.789431 21087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:54:51.789567 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal
uuid: "da3c14bccdb243c5a3509967f48b5fb2"
format_stamp: "Formatted at 2026-08-06 02:54:51 on dist-test-slave-3kk6"
server_key: "615d14abde98aaf42071f52233933dbc"
server_key_iv: "ede800de39009aafa846a53f7642d6a9"
server_key_version: "encryptionkey@0"
I20260806 02:54:51.789685 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:51.804800 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:51.805575 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:51.805794 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:51.806469 21087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:51.807576 21087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:51.807628 21087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.807670 21087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:51.807686 21087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:51.816864 21087 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.131:44041
I20260806 02:54:51.816881 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.131:44041 every 8 connection(s)
I20260806 02:54:51.817847 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:51.827119 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 21087
I20260806 02:54:51.827266 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:51.827561 25098 external_mini_cluster.cc:1428] Setting key 4b773e81f4b280de0a5bdf0819b91796
I20260806 02:54:51.827636 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.819863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:36325 (local address 127.24.130.190:37331)
0806 02:54:51.820017 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.820021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.821034 (+ 1013us) server_negotiation.cc:407] Connection header received
0806 02:54:51.822484 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.822487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.822551 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.822660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:51.823628 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.824228 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.825428 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.825661 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.825779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.827046 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:51.827052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:51.827059 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:51.827075 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:51.827095 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:51.827103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:51.827186 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.827411 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.827485 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:54:51.828802 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.820190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:37331 (local address 127.24.130.131:36325)
0806 02:54:51.820859 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:51.820905 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:51.822251 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:51.822872 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:51.822878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:51.822962 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:51.823476 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.823490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.824380 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.824383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.825287 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.825296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.825830 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.825834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.826010 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.826765 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:54:51.826766 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:51.826809 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:54:51.826877 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:51.827189 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:51.827201 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:51.827210 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:54:51.827409 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:54:51.827705 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
I20260806 02:54:51.830045 21270 heartbeater.cc:347] Connected to a master server at 127.24.130.190:37331
I20260806 02:54:51.830255 21270 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:51.830772 21270 heartbeater.cc:511] Master 127.24.130.190:37331 requested a full tablet report, sending...
I20260806 02:54:51.831748 20652 ts_manager.cc:194] Registered new tserver with Master: da3c14bccdb243c5a3509967f48b5fb2 (127.24.130.131:44041)
I20260806 02:54:51.832330 20652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.130.131:36325
I20260806 02:54:51.841912 25098 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:51.848083 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.844237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.24.130.190:37331)
0806 02:54:51.844430 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.844434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.844518 (+ 84us) server_negotiation.cc:407] Connection header received
0806 02:54:51.844643 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.844645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.844688 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.844755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:51.845041 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.845552 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.846455 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.846683 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.846759 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.847586 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:51.847589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:51.847591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:51.847603 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:51.847621 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:51.847629 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:51.847688 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.847820 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.847871 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 02:54:51.851178 20652 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:51.854149 20652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:51.870196 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.863849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.24.130.130:42771)
0806 02:54:51.864361 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.864374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.864643 (+ 269us) server_negotiation.cc:407] Connection header received
0806 02:54:51.864717 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.864721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.864890 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.865052 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:51.866719 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.867754 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.868937 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.869452 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.869599 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.869796 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.869893 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.870006 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":135,"threads_started":1}
I20260806 02:54:51.870200 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.863159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:44041 (local address 127.0.0.1:56038)
0806 02:54:51.863823 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:51.863880 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:51.864051 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:51.864930 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:51.864936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:51.864963 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:51.865118 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.865124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.866721 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.866724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.868495 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.868502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.869606 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.869610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.869728 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.869756 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:54:51.869827 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:54:51.870020 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":177,"spinlock_wait_cycles":13312,"thread_start_us":129,"threads_started":1}
I20260806 02:54:51.870522 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.862982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:46611 (local address 127.0.0.1:58990)
0806 02:54:51.863823 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:51.863880 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:51.864053 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:51.865202 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:51.865205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:51.865227 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:51.865440 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.865449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.866850 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.866853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.868472 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.868485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.870123 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.870127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.870217 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.870247 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:54:51.870315 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:54:51.870380 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":238,"spinlock_wait_cycles":7552,"thread_start_us":127,"threads_started":1}
I20260806 02:54:51.870196 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.863402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42771 (local address 127.0.0.1:49240)
0806 02:54:51.864432 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:51.864448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:51.864545 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:51.865202 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:51.865205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:51.865233 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:51.865448 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.865454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.867898 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.867901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.868804 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:51.868815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.869586 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.869589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:51.869699 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.869736 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:54:51.869853 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:54:51.870014 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":144,"spinlock_wait_cycles":6784,"thread_start_us":113,"threads_started":1}
I20260806 02:54:51.871006 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.863151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.24.130.129:46611)
0806 02:54:51.864589 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.864602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.864629 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:54:51.864704 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.864709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.864902 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.865052 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:51.865580 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.866725 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.868979 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.869758 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.869883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.870057 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.870726 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.870849 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":131,"threads_started":1}
I20260806 02:54:51.871587 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:51.863871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.24.130.131:44041)
0806 02:54:51.864365 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:51.864376 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:51.864413 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:54:51.864498 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:51.864503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:51.864741 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:51.865251 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:51.865360 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.866530 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.868614 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:51.869450 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:51.870304 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:51.871276 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:51.871332 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:54:51.871428 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":177,"threads_started":1}
I20260806 02:54:51.873751 20818 tablet_service.cc:1467] Processing CreateTablet for tablet ac4b50050fb14493ba2a29b99e41a904 (DEFAULT_TABLE table=test-table [id=fd9427a66cea4e4ca66249506bb9f852]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:51.873805 20997 tablet_service.cc:1467] Processing CreateTablet for tablet ac4b50050fb14493ba2a29b99e41a904 (DEFAULT_TABLE table=test-table [id=fd9427a66cea4e4ca66249506bb9f852]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:51.874310 21183 tablet_service.cc:1467] Processing CreateTablet for tablet ac4b50050fb14493ba2a29b99e41a904 (DEFAULT_TABLE table=test-table [id=fd9427a66cea4e4ca66249506bb9f852]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:51.874919 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b50050fb14493ba2a29b99e41a904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:51.874931 20997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b50050fb14493ba2a29b99e41a904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:51.875290 21183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b50050fb14493ba2a29b99e41a904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:51.881651 21296 tablet_bootstrap.cc:492] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: Bootstrap starting.
I20260806 02:54:51.883203 21298 tablet_bootstrap.cc:492] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Bootstrap starting.
I20260806 02:54:51.883965 21296 tablet_bootstrap.cc:654] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:51.884739 21296 log.cc:824] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:51.884950 21298 tablet_bootstrap.cc:654] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:51.885739 21298 log.cc:824] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:51.885888 21297 tablet_bootstrap.cc:492] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Bootstrap starting.
I20260806 02:54:51.887130 21296 tablet_bootstrap.cc:492] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: No bootstrap required, opened a new log
I20260806 02:54:51.887388 21296 ts_tablet_manager.cc:1397] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:54:51.888275 21297 tablet_bootstrap.cc:654] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:51.889117 21297 log.cc:824] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:51.891345 21297 tablet_bootstrap.cc:492] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: No bootstrap required, opened a new log
I20260806 02:54:51.891346 21298 tablet_bootstrap.cc:492] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: No bootstrap required, opened a new log
I20260806 02:54:51.891556 21298 ts_tablet_manager.cc:1397] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 02:54:51.891556 21297 ts_tablet_manager.cc:1397] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:54:51.891748 21296 raft_consensus.cc:348] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:51.892038 21296 raft_consensus.cc:374] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:51.892119 21296 raft_consensus.cc:729] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e3edf89b4041f894060e72a0418385, State: Initialized, Role: FOLLOWER
I20260806 02:54:51.892663 21296 consensus_queue.cc:260] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:51.893991 21296 ts_tablet_manager.cc:1428] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:54:51.894402 20903 heartbeater.cc:503] Master 127.24.130.190:37331 was elected leader, sending a full tablet report...
I20260806 02:54:51.896271 21298 raft_consensus.cc:348] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:51.896271 21297 raft_consensus.cc:348] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:51.896562 21297 raft_consensus.cc:374] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:51.896581 21298 raft_consensus.cc:374] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:51.896637 21298 raft_consensus.cc:729] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5946b20e5ef446c96086daf748e9d9a, State: Initialized, Role: FOLLOWER
I20260806 02:54:51.896678 21297 raft_consensus.cc:729] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c14bccdb243c5a3509967f48b5fb2, State: Initialized, Role: FOLLOWER
I20260806 02:54:51.897131 21298 consensus_queue.cc:260] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:51.897131 21297 consensus_queue.cc:260] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:51.898082 21080 heartbeater.cc:503] Master 127.24.130.190:37331 was elected leader, sending a full tablet report...
I20260806 02:54:51.898190 21298 ts_tablet_manager.cc:1428] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:54:51.898190 21297 ts_tablet_manager.cc:1428] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:54:51.898339 21270 heartbeater.cc:503] Master 127.24.130.190:37331 was elected leader, sending a full tablet report...
W20260806 02:54:51.920377 20904 tablet.cc:2369] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:54:52.068988 21271 tablet.cc:2369] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:52.084322 21306 raft_consensus.cc:482] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:52.084630 21306 raft_consensus.cc:504] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:52.086014 21306 leader_election.cc:288] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e3edf89b4041f894060e72a0418385 (127.24.130.129:46611), f5946b20e5ef446c96086daf748e9d9a (127.24.130.130:42771)
I20260806 02:54:52.090338 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.086284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:46611 (local address 127.24.130.131:58063)
0806 02:54:52.086676 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:52.086693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:52.086818 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:52.087107 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:52.087113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:52.087136 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:52.087286 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.087294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.088358 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.088361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.089200 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.089210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.089918 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.089923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.090008 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.090026 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:54:52.090124 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:54:52.090194 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":15}
I20260806 02:54:52.090338 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.086492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:60333 (local address 127.24.130.130:42771)
0806 02:54:52.086634 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:52.086640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:52.086851 (+ 211us) server_negotiation.cc:407] Connection header received
0806 02:54:52.087002 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:52.087005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:52.087075 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:52.087175 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:52.087526 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.088089 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.089294 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.089719 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.089816 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.089899 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:52.090132 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 02:54:52.090204 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:54:52.090363 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.086342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42771 (local address 127.24.130.131:60333)
0806 02:54:52.086776 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:52.086790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:52.086878 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:52.087226 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:52.087232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:52.087254 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:52.087402 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.087410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.088200 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.088204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.089169 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.089179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.089846 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.089853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.089961 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.089991 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:54:52.090064 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:54:52.090156 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":26,"thread_start_us":151,"threads_started":1}
I20260806 02:54:52.090384 21304 raft_consensus.cc:482] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:52.090593 21304 raft_consensus.cc:504] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:52.091143 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.086424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.131:58063 (local address 127.24.130.129:46611)
0806 02:54:52.086597 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:52.086602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:52.086814 (+ 212us) server_negotiation.cc:407] Connection header received
0806 02:54:52.086931 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:52.086934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:52.086996 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:52.087069 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:52.087425 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.088226 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.089366 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.089754 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.089896 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.089949 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:52.090943 (+ 994us) server_negotiation.cc:299] Negotiation successful
0806 02:54:52.091032 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:54:52.091251 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b50050fb14493ba2a29b99e41a904" candidate_uuid: "da3c14bccdb243c5a3509967f48b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5946b20e5ef446c96086daf748e9d9a" is_pre_election: true
I20260806 02:54:52.091590 21021 raft_consensus.cc:2454] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c14bccdb243c5a3509967f48b5fb2 in term 0.
I20260806 02:54:52.091733 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b50050fb14493ba2a29b99e41a904" candidate_uuid: "da3c14bccdb243c5a3509967f48b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e3edf89b4041f894060e72a0418385" is_pre_election: true
I20260806 02:54:52.091953 20843 raft_consensus.cc:2454] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3c14bccdb243c5a3509967f48b5fb2 in term 0.
I20260806 02:54:52.092046 21304 leader_election.cc:288] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da3c14bccdb243c5a3509967f48b5fb2 (127.24.130.131:44041), f5946b20e5ef446c96086daf748e9d9a (127.24.130.130:42771)
I20260806 02:54:52.092125 21116 leader_election.cc:302] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c14bccdb243c5a3509967f48b5fb2, f5946b20e5ef446c96086daf748e9d9a; no voters:
I20260806 02:54:52.092990 21306 raft_consensus.cc:2781] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:52.093102 21306 raft_consensus.cc:482] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:52.093163 21306 raft_consensus.cc:3037] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:52.094688 21306 raft_consensus.cc:504] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:52.095728 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b50050fb14493ba2a29b99e41a904" candidate_uuid: "da3c14bccdb243c5a3509967f48b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5946b20e5ef446c96086daf748e9d9a"
I20260806 02:54:52.095877 21021 raft_consensus.cc:3037] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:52.097283 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.092861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:35287 (local address 127.24.130.130:42771)
0806 02:54:52.092987 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:52.092991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:52.093004 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:54:52.093051 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:52.093057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:52.093099 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:52.093165 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:52.093608 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.094292 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.095954 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.096495 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.096588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.096639 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:52.097043 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:54:52.097114 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 02:54:52.097349 21021 raft_consensus.cc:2454] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c14bccdb243c5a3509967f48b5fb2 in term 1.
I20260806 02:54:52.097453 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.092721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.130.129:42399 (local address 127.24.130.131:44041)
0806 02:54:52.092863 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:52.092869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:52.092890 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:54:52.092935 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:52.092938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:52.092986 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:52.093072 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:52.093491 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.094202 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.095906 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.096632 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.096738 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.096801 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:52.097208 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 02:54:52.097296 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:54:52.097728 21116 leader_election.cc:302] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da3c14bccdb243c5a3509967f48b5fb2, f5946b20e5ef446c96086daf748e9d9a; no voters:
I20260806 02:54:52.097858 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.092361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.130:42771 (local address 127.24.130.129:35287)
0806 02:54:52.092649 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:52.092664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:52.092758 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:52.093294 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:52.093298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:52.093319 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:52.093459 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.093467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.094435 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.094441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.095751 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.095763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.096775 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.096778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.096878 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.096913 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:54:52.097480 (+ 567us) client_negotiation.cc:240] Negotiation successful
0806 02:54:52.097643 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":4,"thread_start_us":121,"threads_started":1}
I20260806 02:54:52.097999 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.092230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.131:44041 (local address 127.24.130.129:42399)
0806 02:54:52.092502 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:52.092517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:52.092605 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:52.093184 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:52.093189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:52.093238 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:52.093361 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.093370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.094350 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.094353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.095740 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:52.095755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.096911 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.096916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:52.097045 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.097080 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:54:52.097652 (+ 572us) client_negotiation.cc:240] Negotiation successful
0806 02:54:52.097744 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":96,"thread_start_us":101,"threads_started":1}
I20260806 02:54:52.098505 21306 leader_election.cc:288] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 06e3edf89b4041f894060e72a0418385 (127.24.130.129:46611), f5946b20e5ef446c96086daf748e9d9a (127.24.130.130:42771)
I20260806 02:54:52.098558 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b50050fb14493ba2a29b99e41a904" candidate_uuid: "06e3edf89b4041f894060e72a0418385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5946b20e5ef446c96086daf748e9d9a" is_pre_election: true
I20260806 02:54:52.098621 21306 raft_consensus.cc:2781] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:52.098714 21306 raft_consensus.cc:686] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 1 LEADER]: Becoming Leader. State: Replica: da3c14bccdb243c5a3509967f48b5fb2, State: Running, Role: LEADER
I20260806 02:54:52.098766 21021 raft_consensus.cc:2379] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06e3edf89b4041f894060e72a0418385 in current term 1: Already voted for candidate da3c14bccdb243c5a3509967f48b5fb2 in this term.
I20260806 02:54:52.098884 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b50050fb14493ba2a29b99e41a904" candidate_uuid: "06e3edf89b4041f894060e72a0418385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c14bccdb243c5a3509967f48b5fb2" is_pre_election: true
I20260806 02:54:52.099406 21306 consensus_queue.cc:237] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } }
I20260806 02:54:52.099843 20747 leader_election.cc:302] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06e3edf89b4041f894060e72a0418385; no voters: da3c14bccdb243c5a3509967f48b5fb2, f5946b20e5ef446c96086daf748e9d9a
I20260806 02:54:52.100097 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b50050fb14493ba2a29b99e41a904" candidate_uuid: "da3c14bccdb243c5a3509967f48b5fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e3edf89b4041f894060e72a0418385"
I20260806 02:54:52.100132 21304 raft_consensus.cc:3037] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:52.101549 21304 raft_consensus.cc:2726] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:54:52.103276 20843 raft_consensus.cc:2454] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3c14bccdb243c5a3509967f48b5fb2 in term 1.
I20260806 02:54:52.105157 20651 catalog_manager.cc:5495] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3c14bccdb243c5a3509967f48b5fb2 (127.24.130.131). New cstate: current_term: 1 leader_uuid: "da3c14bccdb243c5a3509967f48b5fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3c14bccdb243c5a3509967f48b5fb2" member_type: VOTER last_known_addr { host: "127.24.130.131" port: 44041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 } health_report { overall_health: UNKNOWN } } }
W20260806 02:54:52.116701 21081 tablet.cc:2369] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:52.168650 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.164933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.24.130.131:44041)
0806 02:54:52.165089 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:52.165096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:52.165114 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:54:52.165203 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:52.165206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:52.165271 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:52.165395 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:52.165643 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.166164 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.167142 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:52.167389 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:52.167504 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:52.167824 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:52.168239 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:52.168376 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:52.168447 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 02:54:52.168498 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:54:52.175220 20843 raft_consensus.cc:1261] T ac4b50050fb14493ba2a29b99e41a904 P 06e3edf89b4041f894060e72a0418385 [term 1 FOLLOWER]: Refusing update from remote peer da3c14bccdb243c5a3509967f48b5fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:52.175180 21021 raft_consensus.cc:1261] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 1 FOLLOWER]: Refusing update from remote peer da3c14bccdb243c5a3509967f48b5fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:52.175932 21306 consensus_queue.cc:1035] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5946b20e5ef446c96086daf748e9d9a" member_type: VOTER last_known_addr { host: "127.24.130.130" port: 42771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:52.176211 21311 consensus_queue.cc:1035] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e3edf89b4041f894060e72a0418385" member_type: VOTER last_known_addr { host: "127.24.130.129" port: 46611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:52.193318 20651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:52.193557 20651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:52.196317 20651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ebfd3e8dc5a49e1afb7b65f23d43708: Sending DeleteTablet for 3 replicas of tablet ac4b50050fb14493ba2a29b99e41a904
I20260806 02:54:52.197249 20818 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4b50050fb14493ba2a29b99e41a904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:52 UTC) from {username='slave'} at 127.0.0.1:58990
I20260806 02:54:52.197377 20997 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4b50050fb14493ba2a29b99e41a904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:52 UTC) from {username='slave'} at 127.0.0.1:49240
I20260806 02:54:52.197849 21183 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4b50050fb14493ba2a29b99e41a904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:52 UTC) from {username='slave'} at 127.0.0.1:56038
I20260806 02:54:52.198182 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 20725
I20260806 02:54:52.198429 21329 tablet_replica.cc:330] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: stopping tablet replica
I20260806 02:54:52.198956 21329 raft_consensus.cc:2229] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:52.199536 21329 raft_consensus.cc:2258] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:52.202576 21329 ts_tablet_manager.cc:1905] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:52.207125 21329 ts_tablet_manager.cc:1918] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:52.207305 21329 log.cc:1196] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-2/wal/wals/ac4b50050fb14493ba2a29b99e41a904
I20260806 02:54:52.207701 21329 ts_tablet_manager.cc:1939] T ac4b50050fb14493ba2a29b99e41a904 P da3c14bccdb243c5a3509967f48b5fb2: Deleting consensus metadata
I20260806 02:54:52.209352 20633 catalog_manager.cc:4841] TS da3c14bccdb243c5a3509967f48b5fb2 (127.24.130.131:44041): tablet ac4b50050fb14493ba2a29b99e41a904 (table test-table [id=fd9427a66cea4e4ca66249506bb9f852]) successfully deleted
I20260806 02:54:52.209884 21327 tablet_replica.cc:330] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: stopping tablet replica
I20260806 02:54:52.210374 21327 raft_consensus.cc:2229] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:52.210889 21327 raft_consensus.cc:2258] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:52.217700 21327 ts_tablet_manager.cc:1905] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:52.220198 21327 ts_tablet_manager.cc:1918] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:52.220340 21327 log.cc:1196] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Deleting WAL directory at /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984624291587-25098-0/minicluster-data/ts-1/wal/wals/ac4b50050fb14493ba2a29b99e41a904
I20260806 02:54:52.220654 21327 ts_tablet_manager.cc:1939] T ac4b50050fb14493ba2a29b99e41a904 P f5946b20e5ef446c96086daf748e9d9a: Deleting consensus metadata
I20260806 02:54:52.222019 20637 catalog_manager.cc:4841] TS f5946b20e5ef446c96086daf748e9d9a (127.24.130.130:42771): tablet ac4b50050fb14493ba2a29b99e41a904 (table test-table [id=fd9427a66cea4e4ca66249506bb9f852]) successfully deleted
W20260806 02:54:52.224496 20635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.130.129:46611 (error 108)
I20260806 02:54:52.224819 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 20906
I20260806 02:54:52.235332 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:52.233956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.129:46611 (local address 127.0.0.1:59006)
0806 02:54:52.235098 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:52.235218 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.129:46611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1038}
W20260806 02:54:52.235736 20635 catalog_manager.cc:4584] TS 06e3edf89b4041f894060e72a0418385 (127.24.130.129:46611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac4b50050fb14493ba2a29b99e41a904: Network error: Client connection negotiation failed: client connection to 127.24.130.129:46611: connect: Connection refused (error 111)
I20260806 02:54:52.240682 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 21087
I20260806 02:54:52.252086 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 20608
2026-08-06T02:54:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1326 ms)
[----------] 6 tests from EncryptionPolicyTest (12032 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:52 dist-test-slave-3kk6 krb5kdc[21333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:52 dist-test-slave-3kk6 krb5kdc[21333](info): set up 2 sockets
Aug 06 02:54:52 dist-test-slave-3kk6 krb5kdc[21333](info): commencing operation
krb5kdc: starting...
W20260806 02:54:53.503576 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:53 dist-test-slave-3kk6 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:54:53.639865 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:53.756228 21349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:53.756567 21349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:53.756667 21349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:53.760787 21349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:53.760857 21349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:53.760879 21349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:53.760905 21349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:53.760921 21349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:53.766207 21349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45147
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:53.767406 21349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:53.768397 21349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:53.776844 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:53.776835 21364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:53.776810 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:53.776904 21349 server_base.cc:1034] running on GCE node
I20260806 02:54:53.777650 21349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:53.778815 21349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:53.780011 21349 hybrid_clock.cc:648] HybridClock initialized: now 1785984893779997 us; error 30 us; skew 500 ppm
Aug 06 02:54:53 dist-test-slave-3kk6 krb5kdc[21333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:53.782990 21349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:54:53.783890 21349 webserver.cc:460] Webserver started at http://127.0.0.1:38479/ using document root <none> and password file <none>
I20260806 02:54:53.784471 21349 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:53.784565 21349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:53.784823 21349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:53.787060 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "99135677fc8c4a9bb3a166565e98543c"
format_stamp: "Formatted at 2026-08-06 02:54:53 on dist-test-slave-3kk6"
server_key: "7936a6ddb3f73c4e1ea4e3d12902fb44"
server_key_iv: "12c82a76894fcefafe8088b08fecc761"
server_key_version: "encryptionkey@0"
I20260806 02:54:53.787655 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "99135677fc8c4a9bb3a166565e98543c"
format_stamp: "Formatted at 2026-08-06 02:54:53 on dist-test-slave-3kk6"
server_key: "7936a6ddb3f73c4e1ea4e3d12902fb44"
server_key_iv: "12c82a76894fcefafe8088b08fecc761"
server_key_version: "encryptionkey@0"
I20260806 02:54:53.791570 21349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:54:53.794067 21372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:53.795399 21349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:54:53.795552 21349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "99135677fc8c4a9bb3a166565e98543c"
format_stamp: "Formatted at 2026-08-06 02:54:53 on dist-test-slave-3kk6"
server_key: "7936a6ddb3f73c4e1ea4e3d12902fb44"
server_key_iv: "12c82a76894fcefafe8088b08fecc761"
server_key_version: "encryptionkey@0"
I20260806 02:54:53.795674 21349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:53.810787 21349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:53.814473 21349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:53.814774 21349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:53.822109 21349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45147
I20260806 02:54:53.822157 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45147 every 8 connection(s)
I20260806 02:54:53.823210 21349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:53.826061 21442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:53.828061 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 21349
I20260806 02:54:53.828217 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:53.828557 25098 external_mini_cluster.cc:1428] Setting key 531c8cf799dd1664348ec9fb0328d16e
I20260806 02:54:53.832962 21442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c: Bootstrap starting.
I20260806 02:54:53.835256 21442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:53.835992 21442 log.cc:824] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:54:53 dist-test-slave-3kk6 krb5kdc[21333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:54:53.837854 21442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c: No bootstrap required, opened a new log
I20260806 02:54:53.841162 21442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99135677fc8c4a9bb3a166565e98543c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45147 } }
I20260806 02:54:53.841427 21442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:53.841552 21442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99135677fc8c4a9bb3a166565e98543c, State: Initialized, Role: FOLLOWER
I20260806 02:54:53.842031 21442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99135677fc8c4a9bb3a166565e98543c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45147 } }
I20260806 02:54:53.842233 21442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:53.842304 21442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:53.842412 21442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:53.843618 21442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99135677fc8c4a9bb3a166565e98543c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45147 } }
I20260806 02:54:53.843979 21442 leader_election.cc:302] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99135677fc8c4a9bb3a166565e98543c; no voters:
I20260806 02:54:53.844317 21442 leader_election.cc:288] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:53.844447 21453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:53.844482 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:53.830050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44515 (local address 127.0.0.1:45147)
0806 02:54:53.830549 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:53.830570 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:53.830612 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:54:53.831395 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:53.831422 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:53.831890 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:53.832257 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:53.833125 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:53.834125 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:53.835059 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:53.835603 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:53.835782 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:53.838480 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:53.838485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:53.838499 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:53.838526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:53.840985 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:53.841580 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:53.841586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:53.841594 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:53.841741 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:53.842126 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:53.842130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:53.842132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:53.842688 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:53.842927 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:53.843214 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:54:53.843459 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
I20260806 02:54:53.844725 21453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [term 1 LEADER]: Becoming Leader. State: Replica: 99135677fc8c4a9bb3a166565e98543c, State: Running, Role: LEADER
I20260806 02:54:53.845088 21453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99135677fc8c4a9bb3a166565e98543c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45147 } }
I20260806 02:54:53.845654 21442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:53.847568 21455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99135677fc8c4a9bb3a166565e98543c. Latest consensus state: current_term: 1 leader_uuid: "99135677fc8c4a9bb3a166565e98543c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99135677fc8c4a9bb3a166565e98543c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45147 } } }
I20260806 02:54:53.847707 21455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:53.848749 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:54:53.848760 21454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99135677fc8c4a9bb3a166565e98543c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99135677fc8c4a9bb3a166565e98543c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45147 } } }
I20260806 02:54:53.848847 21454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c [sys.catalog]: This master's current role is: LEADER
W20260806 02:54:53.850947 21473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:54:53.851066 21473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:54:53.851171 21472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:53.852114 21472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:53.857713 21472 catalog_manager.cc:1347] Generated new cluster ID: a5a6b54d98fb41d7a7e98fd06263d45d
I20260806 02:54:53.858039 21472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:53.861598 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:53.850888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58817 (local address 127.0.0.1:45147)
0806 02:54:53.851016 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:53.851020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:53.851411 (+ 391us) server_negotiation.cc:407] Connection header received
0806 02:54:53.851553 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:53.851556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:53.851604 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:53.851710 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:53.852613 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:53.853371 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:53.854684 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:53.855055 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:53.855136 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:53.857499 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:53.857503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:53.857506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:53.857534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:53.859343 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:53.859973 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:53.859977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:53.859979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:53.860065 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:53.860501 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:53.860504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:53.860506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:53.860731 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:53.860863 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:53.861312 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 02:54:53.861433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:54:53.874336 21472 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:53.875380 21472 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:53.882833 21472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99135677fc8c4a9bb3a166565e98543c: Generated new TSK 0
I20260806 02:54:53.883700 21472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:53.888607 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 21349
W20260806 02:54:53.903879 21331 connection.cc:535] client connection to 127.0.0.1:45147 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45147): Connection reset by peer (error 104)
2026-08-06T02:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1636 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:53 dist-test-slave-3kk6 krb5kdc[21486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:53 dist-test-slave-3kk6 krb5kdc[21486](info): set up 2 sockets
Aug 06 02:54:53 dist-test-slave-3kk6 krb5kdc[21486](info): commencing operation
krb5kdc: starting...
W20260806 02:54:55.120285 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:55 dist-test-slave-3kk6 krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:54:55.258553 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:55.376681 21508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:55.376977 21508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:55.377020 21508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:55.380936 21508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:55.381004 21508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:55.381023 21508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:55.381040 21508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:55.381055 21508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:55.386298 21508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:55.387392 21508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:55.388406 21508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:55.397172 21517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:55.397172 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:55.397202 21516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:55.397476 21508 server_base.cc:1034] running on GCE node
I20260806 02:54:55.398154 21508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:55.399399 21508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:55.400589 21508 hybrid_clock.cc:648] HybridClock initialized: now 1785984895400561 us; error 45 us; skew 500 ppm
Aug 06 02:54:55 dist-test-slave-3kk6 krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:55.403625 21508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:54:55.404523 21508 webserver.cc:460] Webserver started at http://127.0.0.1:39963/ using document root <none> and password file <none>
I20260806 02:54:55.405068 21508 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:55.405133 21508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:55.405311 21508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:55.407560 21508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "a60e25240e214073aaf02fecffdd8de6"
format_stamp: "Formatted at 2026-08-06 02:54:55 on dist-test-slave-3kk6"
server_key: "03db817b9c3b6f7a3bf2d6543052a67c"
server_key_iv: "d0cc7b3b28cab24453c3a1f9224d00c5"
server_key_version: "encryptionkey@0"
I20260806 02:54:55.408107 21508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "a60e25240e214073aaf02fecffdd8de6"
format_stamp: "Formatted at 2026-08-06 02:54:55 on dist-test-slave-3kk6"
server_key: "03db817b9c3b6f7a3bf2d6543052a67c"
server_key_iv: "d0cc7b3b28cab24453c3a1f9224d00c5"
server_key_version: "encryptionkey@0"
I20260806 02:54:55.411945 21508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:54:55.414505 21529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:55.415762 21508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:54:55.415921 21508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "a60e25240e214073aaf02fecffdd8de6"
format_stamp: "Formatted at 2026-08-06 02:54:55 on dist-test-slave-3kk6"
server_key: "03db817b9c3b6f7a3bf2d6543052a67c"
server_key_iv: "d0cc7b3b28cab24453c3a1f9224d00c5"
server_key_version: "encryptionkey@0"
I20260806 02:54:55.416054 21508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:55.436128 21508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:55.441386 21508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:55.441715 21508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:55.452936 21508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41241
I20260806 02:54:55.453238 21601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41241 every 8 connection(s)
I20260806 02:54:55.454183 21508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:55.457645 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 21508
I20260806 02:54:55.457818 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:55.458024 21602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:55.458158 25098 external_mini_cluster.cc:1428] Setting key 29f1ab51b611455011d8fc7e1a788c56
I20260806 02:54:55.466334 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6: Bootstrap starting.
I20260806 02:54:55.469762 21602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:55.470754 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:55.460207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52703 (local address 127.0.0.1:41241)
0806 02:54:55.460704 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:55.460722 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:55.460768 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:54:55.461418 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:55.461441 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:55.461879 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:55.462768 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:55.463169 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:55.464692 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:55.466031 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:55.466763 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:55.467401 (+ 638us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:55.468291 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:55.468300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:55.468317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:55.468823 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:55.468893 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:55.468924 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:55.469535 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:55.469641 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 02:54:55.469789 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":172,"threads_started":1}
I20260806 02:54:55.471199 21602 log.cc:824] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:55.474421 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6: No bootstrap required, opened a new log
I20260806 02:54:55.475699 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:54:55.480131 21602 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60e25240e214073aaf02fecffdd8de6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260806 02:54:55.480571 21602 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:55.480682 21602 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60e25240e214073aaf02fecffdd8de6, State: Initialized, Role: FOLLOWER
I20260806 02:54:55.481201 21602 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60e25240e214073aaf02fecffdd8de6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260806 02:54:55.481391 21602 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:55.481469 21602 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:55.481570 21602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:55.482996 21602 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60e25240e214073aaf02fecffdd8de6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260806 02:54:55.483397 21602 leader_election.cc:302] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60e25240e214073aaf02fecffdd8de6; no voters:
I20260806 02:54:55.483799 21602 leader_election.cc:288] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:55.483835 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:55.478756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46595 (local address 127.0.0.1:41241)
0806 02:54:55.478884 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:55.478889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:55.478904 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:54:55.478941 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:55.478943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:55.478986 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:55.479060 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:55.479484 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:55.480205 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:55.481412 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:55.481760 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:55.481840 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:55.483234 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:55.483238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:55.483241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:55.483254 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:55.483282 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:55.483295 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:55.483354 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:55.483627 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 02:54:55.483668 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:54:55.484272 21617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:55.484522 21617 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [term 1 LEADER]: Becoming Leader. State: Replica: a60e25240e214073aaf02fecffdd8de6, State: Running, Role: LEADER
I20260806 02:54:55.484902 21617 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60e25240e214073aaf02fecffdd8de6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } }
I20260806 02:54:55.485219 21602 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:55.487293 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60e25240e214073aaf02fecffdd8de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60e25240e214073aaf02fecffdd8de6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } } }
I20260806 02:54:55.487442 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:55.487742 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60e25240e214073aaf02fecffdd8de6. Latest consensus state: current_term: 1 leader_uuid: "a60e25240e214073aaf02fecffdd8de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60e25240e214073aaf02fecffdd8de6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41241 } } }
I20260806 02:54:55.487982 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6 [sys.catalog]: This master's current role is: LEADER
W20260806 02:54:55.490223 21636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:54:55.490324 21636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:54:55.490412 21639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:55.491431 21639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:55.497185 21639 catalog_manager.cc:1347] Generated new cluster ID: fbb5f1797e5c451896e96925b259a10c
I20260806 02:54:55.497264 21639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:55.534082 21639 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:55.535157 21639 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:55.541422 21639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60e25240e214073aaf02fecffdd8de6: Generated new TSK 0
I20260806 02:54:55.542160 21639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:55.552400 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 21508
W20260806 02:54:55.568591 21483 connection.cc:535] client connection to 127.0.0.1:41241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41241): Connection reset by peer (error 104)
2026-08-06T02:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1664 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:55 dist-test-slave-3kk6 krb5kdc[21647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:55 dist-test-slave-3kk6 krb5kdc[21647](info): set up 2 sockets
Aug 06 02:54:55 dist-test-slave-3kk6 krb5kdc[21647](info): commencing operation
krb5kdc: starting...
W20260806 02:54:56.773554 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:56 dist-test-slave-3kk6 krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:54:56.913542 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:57.032989 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:57.033308 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:57.033360 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:57.037372 21666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:57.037452 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:57.037472 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:57.037489 21666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:57.037505 21666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:57.043149 21666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:57.044281 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:57.045315 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:57.054176 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:57.054176 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:57.054252 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:57.054557 21666 server_base.cc:1034] running on GCE node
I20260806 02:54:57.055198 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:57.056466 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:57.057657 21666 hybrid_clock.cc:648] HybridClock initialized: now 1785984897057644 us; error 32 us; skew 500 ppm
Aug 06 02:54:57 dist-test-slave-3kk6 krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:57.069144 21666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:54:57.070276 21666 webserver.cc:460] Webserver started at http://127.0.0.1:37193/ using document root <none> and password file <none>
I20260806 02:54:57.070951 21666 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:57.071048 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:57.071300 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:57.073524 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "e90e50f556aa445e8d4207aca6c96da5"
format_stamp: "Formatted at 2026-08-06 02:54:57 on dist-test-slave-3kk6"
server_key: "dbcec8d62a6bc73cbb62a282cb7ecf3e"
server_key_iv: "a6b580643fa145a083238814bc822b01"
server_key_version: "encryptionkey@0"
I20260806 02:54:57.074142 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "e90e50f556aa445e8d4207aca6c96da5"
format_stamp: "Formatted at 2026-08-06 02:54:57 on dist-test-slave-3kk6"
server_key: "dbcec8d62a6bc73cbb62a282cb7ecf3e"
server_key_iv: "a6b580643fa145a083238814bc822b01"
server_key_version: "encryptionkey@0"
I20260806 02:54:57.078127 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:54:57.080754 21681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:57.082034 21666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:54:57.082186 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "e90e50f556aa445e8d4207aca6c96da5"
format_stamp: "Formatted at 2026-08-06 02:54:57 on dist-test-slave-3kk6"
server_key: "dbcec8d62a6bc73cbb62a282cb7ecf3e"
server_key_iv: "a6b580643fa145a083238814bc822b01"
server_key_version: "encryptionkey@0"
I20260806 02:54:57.082314 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:57.087265 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:57.090312 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:57.090533 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:57.098296 21666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33255
I20260806 02:54:57.098332 21754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33255 every 8 connection(s)
I20260806 02:54:57.099393 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:57.101954 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 21666
I20260806 02:54:57.102135 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:57.102494 25098 external_mini_cluster.cc:1428] Setting key f1e4e2fc0041ed16914888a8e154e514
I20260806 02:54:57.102579 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:57.108348 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:57.104111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39165 (local address 127.0.0.1:33255)
0806 02:54:57.104610 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:57.104626 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:57.104663 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:54:57.105447 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:57.105466 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:57.105759 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:57.106228 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:57.107128 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:57.107134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:57.107153 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:57.107165 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:57.107220 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:57.107246 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:57.107397 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:57.107472 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 02:54:57.107632 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":164,"threads_started":1}
I20260806 02:54:57.109568 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5: Bootstrap starting.
I20260806 02:54:57.111162 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:54:57.112087 21755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:57.112890 21755 log.cc:824] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:57.115224 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:57.113339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53539 (local address 127.0.0.1:33255)
0806 02:54:57.113471 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:57.113476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:57.113707 (+ 231us) server_negotiation.cc:407] Connection header received
0806 02:54:57.113859 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:57.113862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:57.113927 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:57.114023 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:57.114852 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:57.114855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:57.114858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:54:57.114870 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:57.114899 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:54:57.114912 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:57.115005 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:57.115066 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 02:54:57.115103 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:54:57.115379 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5: No bootstrap required, opened a new log
I20260806 02:54:57.119179 21755 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90e50f556aa445e8d4207aca6c96da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260806 02:54:57.119454 21755 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:57.119504 21755 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90e50f556aa445e8d4207aca6c96da5, State: Initialized, Role: FOLLOWER
I20260806 02:54:57.119906 21755 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90e50f556aa445e8d4207aca6c96da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260806 02:54:57.120026 21755 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:57.120069 21755 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:57.120144 21755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:57.121150 21755 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90e50f556aa445e8d4207aca6c96da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260806 02:54:57.121483 21755 leader_election.cc:302] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90e50f556aa445e8d4207aca6c96da5; no voters:
I20260806 02:54:57.121872 21755 leader_election.cc:288] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:57.121980 21767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:57.122257 21767 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [term 1 LEADER]: Becoming Leader. State: Replica: e90e50f556aa445e8d4207aca6c96da5, State: Running, Role: LEADER
I20260806 02:54:57.122617 21767 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90e50f556aa445e8d4207aca6c96da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260806 02:54:57.123296 21755 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:57.124430 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90e50f556aa445e8d4207aca6c96da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90e50f556aa445e8d4207aca6c96da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } } }
I20260806 02:54:57.124567 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:57.124918 21769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90e50f556aa445e8d4207aca6c96da5. Latest consensus state: current_term: 1 leader_uuid: "e90e50f556aa445e8d4207aca6c96da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90e50f556aa445e8d4207aca6c96da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } } }
I20260806 02:54:57.124977 21779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:57.125088 21769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:57.128039 21779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:57.134231 21779 catalog_manager.cc:1347] Generated new cluster ID: 81d3fd4e0c054baabe8ac1cfd45fdeb1
I20260806 02:54:57.134330 21779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:57.155437 21779 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:57.155867 21779 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:57.175163 21779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90e50f556aa445e8d4207aca6c96da5: Generated new TSK 0
I20260806 02:54:57.176178 21779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:57.188030 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 21666
2026-08-06T02:54:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:57 dist-test-slave-3kk6 krb5kdc[21797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:57 dist-test-slave-3kk6 krb5kdc[21797](info): set up 2 sockets
Aug 06 02:54:57 dist-test-slave-3kk6 krb5kdc[21797](info): commencing operation
krb5kdc: starting...
W20260806 02:54:58.428190 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:54:58 dist-test-slave-3kk6 krb5kdc[21797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:54:58.569600 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:58.688576 21815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:58.688971 21815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:58.689055 21815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:58.693339 21815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:58.693463 21815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:58.693504 21815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:58.693542 21815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:58.693583 21815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:58.699285 21815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:54:58.700536 21815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:58.701654 21815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:58.711288 21824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:58.711288 21825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:58.711495 21815 server_base.cc:1034] running on GCE node
W20260806 02:54:58.711288 21828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:58.712416 21815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:58.713663 21815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:58.714831 21815 hybrid_clock.cc:648] HybridClock initialized: now 1785984898714798 us; error 51 us; skew 500 ppm
Aug 06 02:54:58 dist-test-slave-3kk6 krb5kdc[21797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:58.717941 21815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:54:58.718925 21815 webserver.cc:460] Webserver started at http://127.0.0.1:35267/ using document root <none> and password file <none>
I20260806 02:54:58.719481 21815 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:58.719542 21815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:58.719720 21815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:54:58.721870 21815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "8216fb2e89f841189e9b8f71ddc4c7eb"
format_stamp: "Formatted at 2026-08-06 02:54:58 on dist-test-slave-3kk6"
server_key: "209bb5d163f59d3ff2eab739b0b8825d"
server_key_iv: "f93ed496c83cf6acbb44a5b8a1621987"
server_key_version: "encryptionkey@0"
I20260806 02:54:58.722432 21815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "8216fb2e89f841189e9b8f71ddc4c7eb"
format_stamp: "Formatted at 2026-08-06 02:54:58 on dist-test-slave-3kk6"
server_key: "209bb5d163f59d3ff2eab739b0b8825d"
server_key_iv: "f93ed496c83cf6acbb44a5b8a1621987"
server_key_version: "encryptionkey@0"
I20260806 02:54:58.726377 21815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:54:58.728983 21836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:58.730265 21815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:54:58.730404 21815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "8216fb2e89f841189e9b8f71ddc4c7eb"
format_stamp: "Formatted at 2026-08-06 02:54:58 on dist-test-slave-3kk6"
server_key: "209bb5d163f59d3ff2eab739b0b8825d"
server_key_iv: "f93ed496c83cf6acbb44a5b8a1621987"
server_key_version: "encryptionkey@0"
I20260806 02:54:58.730522 21815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:58.746025 21815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:58.749877 21815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:58.750124 21815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:58.758085 21815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43821
I20260806 02:54:58.758188 21908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43821 every 8 connection(s)
I20260806 02:54:58.759235 21815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:58.762306 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:58.767894 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 21815
I20260806 02:54:58.768072 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:54:58.768400 25098 external_mini_cluster.cc:1428] Setting key 0ab19ffb49dfb715d8c09d139a92a877
I20260806 02:54:58.770071 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb: Bootstrap starting.
I20260806 02:54:58.772657 21909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:58.773494 21909 log.cc:824] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:58.775682 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb: No bootstrap required, opened a new log
Aug 06 02:54:58 dist-test-slave-3kk6 krb5kdc[21797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:54:58.779894 21909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260806 02:54:58.780262 21909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:58.780340 21909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8216fb2e89f841189e9b8f71ddc4c7eb, State: Initialized, Role: FOLLOWER
I20260806 02:54:58.780907 21909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260806 02:54:58.781098 21909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:58.781178 21909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:58.781291 21909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:58.782680 21909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260806 02:54:58.783082 21909 leader_election.cc:302] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8216fb2e89f841189e9b8f71ddc4c7eb; no voters:
I20260806 02:54:58.783555 21909 leader_election.cc:288] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:58.783669 21916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:58.783941 21916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [term 1 LEADER]: Becoming Leader. State: Replica: 8216fb2e89f841189e9b8f71ddc4c7eb, State: Running, Role: LEADER
I20260806 02:54:58.784437 21916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } }
I20260806 02:54:58.784893 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:58.769992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60995 (local address 127.0.0.1:43821)
0806 02:54:58.770536 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:58.770553 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:58.770593 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:54:58.771383 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:58.771403 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:58.771806 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:58.772184 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:58.773169 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:58.774262 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:58.775263 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:58.775767 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:58.775941 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:58.778889 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:58.778896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:58.778914 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:58.778957 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:58.781214 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:58.781714 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:58.781721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:58.781727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:58.781874 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:58.782158 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:58.782161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:58.782163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:58.782802 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:58.782996 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:58.783283 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:54:58.783630 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":190,"threads_started":1}
I20260806 02:54:58.784910 21909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:58.786736 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } } }
I20260806 02:54:58.786875 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:58.787269 21918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8216fb2e89f841189e9b8f71ddc4c7eb. Latest consensus state: current_term: 1 leader_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216fb2e89f841189e9b8f71ddc4c7eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43821 } } }
I20260806 02:54:58.787918 21934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:58.788036 21918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:58.788576 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:54:58.791268 21934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:58.798136 21934 catalog_manager.cc:1347] Generated new cluster ID: fbbf7662f0dc4bacb7d3a475ada255e9
I20260806 02:54:58.798207 21934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:58.799352 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:58.790691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49597 (local address 127.0.0.1:43821)
0806 02:54:58.790814 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:58.790819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:58.791038 (+ 219us) server_negotiation.cc:407] Connection header received
0806 02:54:58.791199 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:58.791202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:58.791253 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:58.791339 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:58.792084 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:58.792641 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:58.793527 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:58.793866 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:58.793946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:58.795889 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:58.795893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:58.795896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:58.795927 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:58.797542 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:58.798036 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:58.798041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:58.798043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:58.798132 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:58.798422 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:58.798426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:58.798428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:58.798620 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:58.798807 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:58.799041 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 02:54:58.799171 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 02:54:58.806705 21934 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:58.807660 21934 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:58.817010 21934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8216fb2e89f841189e9b8f71ddc4c7eb: Generated new TSK 0
I20260806 02:54:58.817811 21934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:54:58.828176 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 21815
2026-08-06T02:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:58 dist-test-slave-3kk6 krb5kdc[21953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:58 dist-test-slave-3kk6 krb5kdc[21953](info): set up 2 sockets
Aug 06 02:54:58 dist-test-slave-3kk6 krb5kdc[21953](info): commencing operation
krb5kdc: starting...
W20260806 02:55:00.065258 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:00 dist-test-slave-3kk6 krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:00.214775 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:00.336644 21971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:00.337098 21971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:00.337186 21971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:00.341454 21971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:00.341598 21971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:00.341635 21971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:00.341674 21971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:00.341710 21971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:00.347527 21971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:00.348835 21971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:00.350095 21971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:00.359712 21976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:00.359722 21977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:00.359723 21979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:00.359851 21971 server_base.cc:1034] running on GCE node
I20260806 02:55:00.360706 21971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:00.362040 21971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:00.363306 21971 hybrid_clock.cc:648] HybridClock initialized: now 1785984900363319 us; error 201 us; skew 500 ppm
Aug 06 02:55:00 dist-test-slave-3kk6 krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:00.366560 21971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:00.367622 21971 webserver.cc:460] Webserver started at http://127.0.0.1:45269/ using document root <none> and password file <none>
I20260806 02:55:00.368223 21971 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:00.368291 21971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:00.368557 21971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:00.370859 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "aebfdde34c054ca2a5a89a443739586d"
format_stamp: "Formatted at 2026-08-06 02:55:00 on dist-test-slave-3kk6"
server_key: "e2f8c77456d22ee81bcabf574787763d"
server_key_iv: "adfadf3e83945fe926e29c66e52d7502"
server_key_version: "encryptionkey@0"
I20260806 02:55:00.371495 21971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "aebfdde34c054ca2a5a89a443739586d"
format_stamp: "Formatted at 2026-08-06 02:55:00 on dist-test-slave-3kk6"
server_key: "e2f8c77456d22ee81bcabf574787763d"
server_key_iv: "adfadf3e83945fe926e29c66e52d7502"
server_key_version: "encryptionkey@0"
I20260806 02:55:00.375828 21971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:00.378635 21986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:00.380173 21971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:55:00.380352 21971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "aebfdde34c054ca2a5a89a443739586d"
format_stamp: "Formatted at 2026-08-06 02:55:00 on dist-test-slave-3kk6"
server_key: "e2f8c77456d22ee81bcabf574787763d"
server_key_iv: "adfadf3e83945fe926e29c66e52d7502"
server_key_version: "encryptionkey@0"
I20260806 02:55:00.380498 21971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:00.398608 21971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:00.402438 21971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:00.402711 21971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:00.410554 21971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35345
I20260806 02:55:00.410570 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35345 every 8 connection(s)
I20260806 02:55:00.411661 21971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:00.414741 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:00.415084 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 21971
I20260806 02:55:00.415251 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:00.415556 25098 external_mini_cluster.cc:1428] Setting key c8d2ed5e7cf804c231e0957d6dad5c17
I20260806 02:55:00.422562 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d: Bootstrap starting.
I20260806 02:55:00.425592 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:00.426352 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:00.417012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39233 (local address 127.0.0.1:35345)
0806 02:55:00.417506 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:00.417526 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:00.417573 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:55:00.418388 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:00.418414 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:00.418933 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:00.419326 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:00.419804 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:00.421356 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:00.422321 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:00.423077 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:00.423251 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:00.424185 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:00.424192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:00.424210 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:00.424744 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:00.424818 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:00.424850 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:00.424951 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:00.425068 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 02:55:00.425293 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":158,"threads_started":1}
I20260806 02:55:00.426570 22057 log.cc:824] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:00.428757 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d: No bootstrap required, opened a new log
I20260806 02:55:00.429334 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:00.433732 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfdde34c054ca2a5a89a443739586d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260806 02:55:00.434002 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:00.434041 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebfdde34c054ca2a5a89a443739586d, State: Initialized, Role: FOLLOWER
I20260806 02:55:00.434561 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfdde34c054ca2a5a89a443739586d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260806 02:55:00.434754 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:00.434840 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:00.434957 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:00.436406 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:00.431782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49677 (local address 127.0.0.1:35345)
0806 02:55:00.431963 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:00.431968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:00.432149 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:55:00.432245 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:00.432248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:00.432296 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:00.432394 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:00.432702 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:00.433469 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:00.434455 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:00.434862 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:00.434982 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:00.435991 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:00.435994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:00.435996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:00.436008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:00.436036 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:00.436046 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:00.436108 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:00.436244 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 02:55:00.436300 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:55:00.436399 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfdde34c054ca2a5a89a443739586d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260806 02:55:00.436800 22057 leader_election.cc:302] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aebfdde34c054ca2a5a89a443739586d; no voters:
I20260806 02:55:00.437196 22057 leader_election.cc:288] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:00.437306 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:00.437664 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [term 1 LEADER]: Becoming Leader. State: Replica: aebfdde34c054ca2a5a89a443739586d, State: Running, Role: LEADER
I20260806 02:55:00.438022 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfdde34c054ca2a5a89a443739586d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } }
I20260806 02:55:00.438578 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:00.439234 22072 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aebfdde34c054ca2a5a89a443739586d. Latest consensus state: current_term: 1 leader_uuid: "aebfdde34c054ca2a5a89a443739586d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfdde34c054ca2a5a89a443739586d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } } }
I20260806 02:55:00.439371 22072 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:00.439736 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aebfdde34c054ca2a5a89a443739586d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebfdde34c054ca2a5a89a443739586d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35345 } } }
I20260806 02:55:00.439885 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:00.439937 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:00.443862 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:00.450435 22082 catalog_manager.cc:1347] Generated new cluster ID: 9809407e958c4544b42710104b5fee52
I20260806 02:55:00.450536 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:00.469676 22082 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:00.470911 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:00.480849 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aebfdde34c054ca2a5a89a443739586d: Generated new TSK 0
I20260806 02:55:00.481907 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:00.494776 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 21971
2026-08-06T02:55:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1666 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:00 dist-test-slave-3kk6 krb5kdc[22099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:00 dist-test-slave-3kk6 krb5kdc[22099](info): set up 2 sockets
Aug 06 02:55:00 dist-test-slave-3kk6 krb5kdc[22099](info): commencing operation
krb5kdc: starting...
W20260806 02:55:01.743276 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:01 dist-test-slave-3kk6 krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:01.886255 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:02.007884 22118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:02.008239 22118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:02.008294 22118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:02.012626 22118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:02.012734 22118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:02.012755 22118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:02.012773 22118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:02.012789 22118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:02.018700 22118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:02.020006 22118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:02.021166 22118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:02.030875 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:02.030875 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:02.030875 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:02.031386 22118 server_base.cc:1034] running on GCE node
I20260806 02:55:02.032068 22118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:02.033294 22118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:02.034467 22118 hybrid_clock.cc:648] HybridClock initialized: now 1785984902034447 us; error 36 us; skew 500 ppm
Aug 06 02:55:02 dist-test-slave-3kk6 krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:02.037691 22118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:02.038822 22118 webserver.cc:460] Webserver started at http://127.0.0.1:42537/ using document root <none> and password file <none>
I20260806 02:55:02.039433 22118 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:02.039501 22118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:02.039690 22118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:02.042004 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "63596e4e03154688bd9a729c2759de5f"
format_stamp: "Formatted at 2026-08-06 02:55:02 on dist-test-slave-3kk6"
server_key: "d780fe90e415f8896c6de57145d77025"
server_key_iv: "6002e8839f02ab5a734f7a6dccf87724"
server_key_version: "encryptionkey@0"
I20260806 02:55:02.042629 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "63596e4e03154688bd9a729c2759de5f"
format_stamp: "Formatted at 2026-08-06 02:55:02 on dist-test-slave-3kk6"
server_key: "d780fe90e415f8896c6de57145d77025"
server_key_iv: "6002e8839f02ab5a734f7a6dccf87724"
server_key_version: "encryptionkey@0"
I20260806 02:55:02.047118 22118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:55:02.050053 22138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:02.051604 22118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:02.051765 22118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "63596e4e03154688bd9a729c2759de5f"
format_stamp: "Formatted at 2026-08-06 02:55:02 on dist-test-slave-3kk6"
server_key: "d780fe90e415f8896c6de57145d77025"
server_key_iv: "6002e8839f02ab5a734f7a6dccf87724"
server_key_version: "encryptionkey@0"
I20260806 02:55:02.051898 22118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:02.057351 22118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:02.061332 22118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:02.061524 22118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:02.069487 22118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33565
I20260806 02:55:02.069515 22213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33565 every 8 connection(s)
I20260806 02:55:02.070544 22118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:02.073626 22214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:02.075038 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 22118
I20260806 02:55:02.075204 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:02.075520 25098 external_mini_cluster.cc:1428] Setting key fdaad4bace3fd2a34647cf5b6ffd5a0f
I20260806 02:55:02.080345 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f: Bootstrap starting.
I20260806 02:55:02.082803 22214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:02.083679 22214 log.cc:824] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:02.084860 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:02.077129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52733 (local address 127.0.0.1:33565)
0806 02:55:02.077620 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:02.077636 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:02.077674 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:55:02.078353 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:02.078371 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:02.078728 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:02.079079 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:02.083457 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:02.083462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:02.083477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:02.083492 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:02.083556 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:02.083578 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:02.083717 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:02.083831 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:55:02.084016 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":151,"threads_started":1}
I20260806 02:55:02.086127 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f: No bootstrap required, opened a new log
I20260806 02:55:02.088022 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:02.090544 22214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63596e4e03154688bd9a729c2759de5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260806 02:55:02.090950 22214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:02.091032 22214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63596e4e03154688bd9a729c2759de5f, State: Initialized, Role: FOLLOWER
I20260806 02:55:02.091564 22214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63596e4e03154688bd9a729c2759de5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260806 02:55:02.091777 22214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:02.091861 22214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:02.091969 22214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:02.092350 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:02.090292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60891 (local address 127.0.0.1:33565)
0806 02:55:02.090444 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:02.090449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:02.090694 (+ 245us) server_negotiation.cc:407] Connection header received
0806 02:55:02.090846 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:02.090849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:02.090893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:02.090986 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:02.091795 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:02.091800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:02.091802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:02.091814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:02.091848 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:02.091877 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:02.091964 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:02.092169 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 02:55:02.092215 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:55:02.093295 22214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63596e4e03154688bd9a729c2759de5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260806 02:55:02.093678 22214 leader_election.cc:302] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63596e4e03154688bd9a729c2759de5f; no voters:
I20260806 02:55:02.094040 22214 leader_election.cc:288] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:02.094163 22227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:02.094434 22227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [term 1 LEADER]: Becoming Leader. State: Replica: 63596e4e03154688bd9a729c2759de5f, State: Running, Role: LEADER
I20260806 02:55:02.094805 22227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63596e4e03154688bd9a729c2759de5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } }
I20260806 02:55:02.096838 22228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63596e4e03154688bd9a729c2759de5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63596e4e03154688bd9a729c2759de5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } } }
I20260806 02:55:02.097059 22228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:02.097549 22230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63596e4e03154688bd9a729c2759de5f. Latest consensus state: current_term: 1 leader_uuid: "63596e4e03154688bd9a729c2759de5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63596e4e03154688bd9a729c2759de5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33565 } } }
I20260806 02:55:02.097663 22230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:02.100826 22214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:02.101047 22234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:02.104777 22234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:02.111583 22234 catalog_manager.cc:1347] Generated new cluster ID: 3f1a9fec366748a6b46b632db3565e65
I20260806 02:55:02.111701 22234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:02.135653 22234 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:02.136075 22234 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:02.144313 22234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63596e4e03154688bd9a729c2759de5f: Generated new TSK 0
I20260806 02:55:02.145117 22234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:02.156790 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 22118
2026-08-06T02:55:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1661 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:02 dist-test-slave-3kk6 krb5kdc[22252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:02 dist-test-slave-3kk6 krb5kdc[22252](info): set up 2 sockets
Aug 06 02:55:02 dist-test-slave-3kk6 krb5kdc[22252](info): commencing operation
krb5kdc: starting...
W20260806 02:55:03.385998 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:03 dist-test-slave-3kk6 krb5kdc[22252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:03.526888 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:03.648465 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:03.648801 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:03.648846 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:03.652972 22271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:03.653066 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:03.653087 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:03.653105 22271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:03.653120 22271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:03.658990 22271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:03.660176 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:03.661233 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:03.670223 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:03.670226 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:03.670228 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:03.670495 22271 server_base.cc:1034] running on GCE node
I20260806 02:55:03.671188 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:03.672416 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:03.673594 22271 hybrid_clock.cc:648] HybridClock initialized: now 1785984903673576 us; error 37 us; skew 500 ppm
Aug 06 02:55:03 dist-test-slave-3kk6 krb5kdc[22252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:03.676661 22271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:03.677736 22271 webserver.cc:460] Webserver started at http://127.0.0.1:35243/ using document root <none> and password file <none>
I20260806 02:55:03.678332 22271 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:03.678435 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:03.678728 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:03.680951 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "1b98c760f0954ca69bd97dc59a808b04"
format_stamp: "Formatted at 2026-08-06 02:55:03 on dist-test-slave-3kk6"
server_key: "1953bf8c42a1952ffd7a78a53ad7508c"
server_key_iv: "f080c6584e2c993f18c8488956f1cc74"
server_key_version: "encryptionkey@0"
I20260806 02:55:03.681589 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "1b98c760f0954ca69bd97dc59a808b04"
format_stamp: "Formatted at 2026-08-06 02:55:03 on dist-test-slave-3kk6"
server_key: "1953bf8c42a1952ffd7a78a53ad7508c"
server_key_iv: "f080c6584e2c993f18c8488956f1cc74"
server_key_version: "encryptionkey@0"
I20260806 02:55:03.685603 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 02:55:03.688328 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:03.689613 22271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 02:55:03.689769 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "1b98c760f0954ca69bd97dc59a808b04"
format_stamp: "Formatted at 2026-08-06 02:55:03 on dist-test-slave-3kk6"
server_key: "1953bf8c42a1952ffd7a78a53ad7508c"
server_key_iv: "f080c6584e2c993f18c8488956f1cc74"
server_key_version: "encryptionkey@0"
I20260806 02:55:03.689899 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:03.708997 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:03.712621 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:03.712865 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:03.720861 22271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46349
I20260806 02:55:03.720872 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46349 every 8 connection(s)
I20260806 02:55:03.721926 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:03.724862 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:03.725993 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 22271
I20260806 02:55:03.726153 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:03.726440 25098 external_mini_cluster.cc:1428] Setting key 337995a6688bbf05d750528f10fd7aa6
I20260806 02:55:03.731835 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04: Bootstrap starting.
I20260806 02:55:03.734737 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:03.735801 22369 log.cc:824] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:55:03 dist-test-slave-3kk6 krb5kdc[22252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:55:03.738059 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04: No bootstrap required, opened a new log
I20260806 02:55:03.741945 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b98c760f0954ca69bd97dc59a808b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260806 02:55:03.742245 22369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:03.742339 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b98c760f0954ca69bd97dc59a808b04, State: Initialized, Role: FOLLOWER
I20260806 02:55:03.742730 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:03.728198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:33941 (local address 127.0.0.1:46349)
0806 02:55:03.728659 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:03.728677 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:03.728718 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:55:03.729386 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:03.729403 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:03.729764 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:03.730056 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:03.731088 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:03.732185 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:03.733368 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:03.733843 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:03.733971 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:03.737189 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:03.737195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:03.737210 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:03.737240 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:03.739234 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:03.739895 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:03.739900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:03.739905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:03.740030 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:03.740420 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:03.740422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:03.740424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:03.740884 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:03.741149 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:03.741478 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:55:03.741729 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":157,"threads_started":1}
I20260806 02:55:03.742882 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b98c760f0954ca69bd97dc59a808b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260806 02:55:03.743062 22369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:03.743151 22369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:03.743256 22369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:03.744570 22369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b98c760f0954ca69bd97dc59a808b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260806 02:55:03.744971 22369 leader_election.cc:302] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b98c760f0954ca69bd97dc59a808b04; no voters:
I20260806 02:55:03.745424 22369 leader_election.cc:288] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:03.745546 22375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:03.745820 22375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [term 1 LEADER]: Becoming Leader. State: Replica: 1b98c760f0954ca69bd97dc59a808b04, State: Running, Role: LEADER
I20260806 02:55:03.746150 22375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b98c760f0954ca69bd97dc59a808b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260806 02:55:03.746320 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:03.746917 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:03.747493 22377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b98c760f0954ca69bd97dc59a808b04. Latest consensus state: current_term: 1 leader_uuid: "1b98c760f0954ca69bd97dc59a808b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b98c760f0954ca69bd97dc59a808b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } } }
I20260806 02:55:03.747640 22377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:03.747985 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b98c760f0954ca69bd97dc59a808b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b98c760f0954ca69bd97dc59a808b04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } } }
I20260806 02:55:03.748081 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04 [sys.catalog]: This master's current role is: LEADER
W20260806 02:55:03.752110 22397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:03.752213 22397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:03.752305 22385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:03.753253 22385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:03.759231 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:03.750140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:50703 (local address 127.0.0.1:46349)
0806 02:55:03.750250 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:03.750255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:03.750449 (+ 194us) server_negotiation.cc:407] Connection header received
0806 02:55:03.750688 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:03.750692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:03.750749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:03.750841 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:03.751763 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:03.752536 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:03.753484 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:03.753738 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:03.753853 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:03.755492 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:03.755496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:03.755499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:03.755526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:03.757048 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:03.757552 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:03.757556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:03.757558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:03.757640 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:03.758095 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:03.758099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:03.758100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:03.758330 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:03.758483 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:03.758963 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:55:03.759064 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 02:55:03.760401 22385 catalog_manager.cc:1347] Generated new cluster ID: 1c6e78097ac14d81aac71363542cd0d4
I20260806 02:55:03.760475 22385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:03.773558 22385 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:03.774647 22385 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:03.780687 22385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b98c760f0954ca69bd97dc59a808b04: Generated new TSK 0
I20260806 02:55:03.781476 22385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:03.787672 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 22271
W20260806 02:55:03.800529 22250 connection.cc:535] client connection to 127.0.0.1:46349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46349): Connection reset by peer (error 104)
2026-08-06T02:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:03 dist-test-slave-3kk6 krb5kdc[22407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:03 dist-test-slave-3kk6 krb5kdc[22407](info): set up 2 sockets
Aug 06 02:55:03 dist-test-slave-3kk6 krb5kdc[22407](info): commencing operation
krb5kdc: starting...
W20260806 02:55:05.023233 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:05 dist-test-slave-3kk6 krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:05.166709 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:05.287838 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:05.288172 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:05.288216 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:05.292543 22423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:05.292641 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:05.292662 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:05.292680 22423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:05.292696 22423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:05.298295 22423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:05.299522 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:05.300614 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:05.310216 22434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:05.310236 22430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:05.310209 22429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:05.310698 22423 server_base.cc:1034] running on GCE node
I20260806 02:55:05.311395 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:05.312765 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:05.313992 22423 hybrid_clock.cc:648] HybridClock initialized: now 1785984905313979 us; error 42 us; skew 500 ppm
Aug 06 02:55:05 dist-test-slave-3kk6 krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:05.317304 22423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:05.318295 22423 webserver.cc:460] Webserver started at http://127.0.0.1:42225/ using document root <none> and password file <none>
I20260806 02:55:05.318964 22423 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:05.319038 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:05.319228 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:05.321521 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "61ce211e9f40473584ba1a86cc3d267d"
format_stamp: "Formatted at 2026-08-06 02:55:05 on dist-test-slave-3kk6"
server_key: "725aac56dccd0c5c697e1ae26eb3fab3"
server_key_iv: "d1a7cd40216e2bcddfc03df11ad9b0ad"
server_key_version: "encryptionkey@0"
I20260806 02:55:05.322130 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "61ce211e9f40473584ba1a86cc3d267d"
format_stamp: "Formatted at 2026-08-06 02:55:05 on dist-test-slave-3kk6"
server_key: "725aac56dccd0c5c697e1ae26eb3fab3"
server_key_iv: "d1a7cd40216e2bcddfc03df11ad9b0ad"
server_key_version: "encryptionkey@0"
I20260806 02:55:05.326464 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:55:05.329270 22443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:05.330757 22423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:05.330891 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "61ce211e9f40473584ba1a86cc3d267d"
format_stamp: "Formatted at 2026-08-06 02:55:05 on dist-test-slave-3kk6"
server_key: "725aac56dccd0c5c697e1ae26eb3fab3"
server_key_iv: "d1a7cd40216e2bcddfc03df11ad9b0ad"
server_key_version: "encryptionkey@0"
I20260806 02:55:05.330997 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:05.358937 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:05.362702 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:05.362926 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:05.371322 22423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36809
I20260806 02:55:05.371328 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36809 every 8 connection(s)
I20260806 02:55:05.372496 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:05.375588 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:05.375653 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 22423
I20260806 02:55:05.375871 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:05.376204 25098 external_mini_cluster.cc:1428] Setting key 5870867cf6e72676435430c84499d099
I20260806 02:55:05.382875 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d: Bootstrap starting.
I20260806 02:55:05.385629 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:05.386392 22522 log.cc:824] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:05.386554 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:05.377689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:33795 (local address 127.0.0.1:36809)
0806 02:55:05.378168 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:05.378185 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:05.378240 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:55:05.379097 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:05.379119 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:05.379517 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:05.379832 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:05.380263 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:05.381363 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:05.382625 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:05.383280 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:05.383422 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:05.384513 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:05.384518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:05.384532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:05.384965 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:05.385024 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:05.385048 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:05.385167 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:05.385540 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 02:55:05.385701 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":159,"threads_started":1}
I20260806 02:55:05.388746 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d: No bootstrap required, opened a new log
I20260806 02:55:05.389672 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:05.393015 22522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ce211e9f40473584ba1a86cc3d267d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260806 02:55:05.393297 22522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:05.393397 22522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ce211e9f40473584ba1a86cc3d267d, State: Initialized, Role: FOLLOWER
I20260806 02:55:05.393882 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ce211e9f40473584ba1a86cc3d267d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260806 02:55:05.394042 22522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:05.394133 22522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:05.394237 22522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:05.395386 22522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ce211e9f40473584ba1a86cc3d267d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260806 02:55:05.395787 22522 leader_election.cc:302] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61ce211e9f40473584ba1a86cc3d267d; no voters:
I20260806 02:55:05.396173 22522 leader_election.cc:288] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:05.396294 22536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:05.396476 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:05.391924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:41261 (local address 127.0.0.1:36809)
0806 02:55:05.392068 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:05.392073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:05.392265 (+ 192us) server_negotiation.cc:407] Connection header received
0806 02:55:05.392421 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:05.392424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:05.392478 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:05.392566 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:05.392855 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:05.393494 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:05.394500 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:05.394872 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:05.394995 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:05.395901 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:05.395904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:05.395907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:05.395921 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:05.395960 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:05.395978 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:05.396068 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:05.396216 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:55:05.396254 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260806 02:55:05.396539 22536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [term 1 LEADER]: Becoming Leader. State: Replica: 61ce211e9f40473584ba1a86cc3d267d, State: Running, Role: LEADER
I20260806 02:55:05.396883 22536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ce211e9f40473584ba1a86cc3d267d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260806 02:55:05.397593 22522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:05.398126 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61ce211e9f40473584ba1a86cc3d267d. Latest consensus state: current_term: 1 leader_uuid: "61ce211e9f40473584ba1a86cc3d267d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ce211e9f40473584ba1a86cc3d267d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } } }
I20260806 02:55:05.398284 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:05.398118 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61ce211e9f40473584ba1a86cc3d267d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ce211e9f40473584ba1a86cc3d267d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } } }
I20260806 02:55:05.398458 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:05.400153 22543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:05.403000 22543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:05.408242 22543 catalog_manager.cc:1347] Generated new cluster ID: ee51100c98e442c3a82f694fbfdb075a
I20260806 02:55:05.408319 22543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:05.423636 22543 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:05.424841 22543 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:05.433735 22543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61ce211e9f40473584ba1a86cc3d267d: Generated new TSK 0
I20260806 02:55:05.434665 22543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:05.447618 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 22423
W20260806 02:55:05.461074 22403 connection.cc:535] client connection to 127.0.0.1:36809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36809): Connection reset by peer (error 104)
2026-08-06T02:55:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:05 dist-test-slave-3kk6 krb5kdc[22564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:05 dist-test-slave-3kk6 krb5kdc[22564](info): set up 2 sockets
Aug 06 02:55:05 dist-test-slave-3kk6 krb5kdc[22564](info): commencing operation
krb5kdc: starting...
W20260806 02:55:06.674248 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:06 dist-test-slave-3kk6 krb5kdc[22564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:06.820297 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:06.942706 22588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:06.943048 22588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:06.943095 22588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:06.947239 22588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:06.947320 22588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:06.947341 22588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:06.947359 22588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:06.947376 22588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:06.953090 22588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:06.954248 22588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:06.955320 22588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:06.964432 22596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:06.964474 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:06.964432 22594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:06.964886 22588 server_base.cc:1034] running on GCE node
I20260806 02:55:06.965548 22588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:06.966859 22588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:06.968073 22588 hybrid_clock.cc:648] HybridClock initialized: now 1785984906968039 us; error 49 us; skew 500 ppm
Aug 06 02:55:06 dist-test-slave-3kk6 krb5kdc[22564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:06.971186 22588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:06.972254 22588 webserver.cc:460] Webserver started at http://127.0.0.1:44303/ using document root <none> and password file <none>
I20260806 02:55:06.972848 22588 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:06.972949 22588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:06.973201 22588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:06.975497 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "22557e07ec6b4ad4920bb1584d69ecc0"
format_stamp: "Formatted at 2026-08-06 02:55:06 on dist-test-slave-3kk6"
server_key: "fabb16bf8904fe2ef239912ea232d374"
server_key_iv: "e4e0de1a1ba8166f3c8221aa6a27a8aa"
server_key_version: "encryptionkey@0"
I20260806 02:55:06.976117 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "22557e07ec6b4ad4920bb1584d69ecc0"
format_stamp: "Formatted at 2026-08-06 02:55:06 on dist-test-slave-3kk6"
server_key: "fabb16bf8904fe2ef239912ea232d374"
server_key_iv: "e4e0de1a1ba8166f3c8221aa6a27a8aa"
server_key_version: "encryptionkey@0"
I20260806 02:55:06.980254 22588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:06.982831 22612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:06.984112 22588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:55:06.984263 22588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "22557e07ec6b4ad4920bb1584d69ecc0"
format_stamp: "Formatted at 2026-08-06 02:55:06 on dist-test-slave-3kk6"
server_key: "fabb16bf8904fe2ef239912ea232d374"
server_key_iv: "e4e0de1a1ba8166f3c8221aa6a27a8aa"
server_key_version: "encryptionkey@0"
I20260806 02:55:06.984386 22588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:06.989457 22588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:06.993088 22588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:06.993309 22588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:07.001328 22588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40605
I20260806 02:55:07.001355 22684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40605 every 8 connection(s)
I20260806 02:55:07.002404 22588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:07.005448 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:07.008911 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 22588
I20260806 02:55:07.009064 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:07.009356 25098 external_mini_cluster.cc:1428] Setting key d0913c95a32ed404d813bb048818f95e
I20260806 02:55:07.012094 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0: Bootstrap starting.
I20260806 02:55:07.015107 22685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:07.015617 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:07.010977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:33991 (local address 127.0.0.1:40605)
0806 02:55:07.011466 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:07.011485 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:07.011529 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:55:07.012420 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:07.012444 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:07.012816 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:07.013215 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:07.014090 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:07.014097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:07.014112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:07.014124 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:07.014197 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:07.014224 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:07.014366 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:07.014438 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 02:55:07.014593 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260806 02:55:07.016041 22685 log.cc:824] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:07.018388 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0: No bootstrap required, opened a new log
I20260806 02:55:07.018888 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:07.023277 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:07.021195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:39447 (local address 127.0.0.1:40605)
0806 02:55:07.021337 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:07.021343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:07.021545 (+ 202us) server_negotiation.cc:407] Connection header received
0806 02:55:07.021697 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:07.021699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:07.021734 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:07.021855 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:07.022720 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:07.022723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:07.022725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:07.022737 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:07.022765 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:07.022799 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:07.022876 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:07.022994 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 02:55:07.023033 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:55:07.023111 22685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260806 02:55:07.023473 22685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:07.023536 22685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22557e07ec6b4ad4920bb1584d69ecc0, State: Initialized, Role: FOLLOWER
I20260806 02:55:07.024045 22685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260806 02:55:07.024192 22685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:07.024286 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:07.024417 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:07.025529 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260806 02:55:07.025926 22685 leader_election.cc:302] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22557e07ec6b4ad4920bb1584d69ecc0; no voters:
I20260806 02:55:07.026316 22685 leader_election.cc:288] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:07.026482 22700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:07.026767 22700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [term 1 LEADER]: Becoming Leader. State: Replica: 22557e07ec6b4ad4920bb1584d69ecc0, State: Running, Role: LEADER
I20260806 02:55:07.027169 22700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260806 02:55:07.027927 22685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:07.029127 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } } }
I20260806 02:55:07.029286 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:07.029685 22714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:07.030200 22703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22557e07ec6b4ad4920bb1584d69ecc0. Latest consensus state: current_term: 1 leader_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22557e07ec6b4ad4920bb1584d69ecc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } } }
I20260806 02:55:07.030390 22703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:07.033000 22714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:07.039544 22714 catalog_manager.cc:1347] Generated new cluster ID: cea114a4d7b041b3b4e25813e9097ed8
I20260806 02:55:07.039659 22714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:07.070971 22714 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:07.071363 22714 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:07.077160 22714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22557e07ec6b4ad4920bb1584d69ecc0: Generated new TSK 0
I20260806 02:55:07.077967 22714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:07.087894 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 22588
2026-08-06T02:55:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1638 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:07 dist-test-slave-3kk6 krb5kdc[22731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:07 dist-test-slave-3kk6 krb5kdc[22731](info): set up 2 sockets
Aug 06 02:55:07 dist-test-slave-3kk6 krb5kdc[22731](info): commencing operation
krb5kdc: starting...
W20260806 02:55:08.333438 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:08 dist-test-slave-3kk6 krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:08.474056 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:08.594586 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:08.594961 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:08.595010 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:08.599380 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:08.599471 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:08.599493 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:08.599510 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:08.599527 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:08.605324 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:08.606598 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:08.607795 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:08.617040 22757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:08.617085 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:08.617223 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:08.617293 22748 server_base.cc:1034] running on GCE node
I20260806 02:55:08.618000 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:08.619279 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:08.620497 22748 hybrid_clock.cc:648] HybridClock initialized: now 1785984908620472 us; error 44 us; skew 500 ppm
Aug 06 02:55:08 dist-test-slave-3kk6 krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:08.623692 22748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:08.624776 22748 webserver.cc:460] Webserver started at http://127.0.0.1:36513/ using document root <none> and password file <none>
I20260806 02:55:08.625411 22748 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:08.625517 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:08.625790 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:08.628401 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "863222b0fb1e4c3ba85d67313bd0bf56"
format_stamp: "Formatted at 2026-08-06 02:55:08 on dist-test-slave-3kk6"
server_key: "fc5c5125fe11555afe91443a69a3cceb"
server_key_iv: "190f28e8627f9bf3a76c83ceab1ef8af"
server_key_version: "encryptionkey@0"
I20260806 02:55:08.629086 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "863222b0fb1e4c3ba85d67313bd0bf56"
format_stamp: "Formatted at 2026-08-06 02:55:08 on dist-test-slave-3kk6"
server_key: "fc5c5125fe11555afe91443a69a3cceb"
server_key_iv: "190f28e8627f9bf3a76c83ceab1ef8af"
server_key_version: "encryptionkey@0"
I20260806 02:55:08.633279 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:55:08.635944 22770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:08.637223 22748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:55:08.637379 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "863222b0fb1e4c3ba85d67313bd0bf56"
format_stamp: "Formatted at 2026-08-06 02:55:08 on dist-test-slave-3kk6"
server_key: "fc5c5125fe11555afe91443a69a3cceb"
server_key_iv: "190f28e8627f9bf3a76c83ceab1ef8af"
server_key_version: "encryptionkey@0"
I20260806 02:55:08.637511 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:08.652566 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:08.656407 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:08.656652 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:08.664177 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37859
I20260806 02:55:08.664223 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37859 every 8 connection(s)
I20260806 02:55:08.665232 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:08.668329 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:08.672469 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 22748
I20260806 02:55:08.672650 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:08.672972 25098 external_mini_cluster.cc:1428] Setting key d6767b0fd43b7f70d4bb6e104389e6c1
I20260806 02:55:08.674772 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56: Bootstrap starting.
I20260806 02:55:08.677006 22837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:08.677771 22837 log.cc:824] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:08.679814 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56: No bootstrap required, opened a new log
Aug 06 02:55:08 dist-test-slave-3kk6 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:55:08.684044 22837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260806 02:55:08.684314 22837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:08.684401 22837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863222b0fb1e4c3ba85d67313bd0bf56, State: Initialized, Role: FOLLOWER
I20260806 02:55:08.684914 22837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260806 02:55:08.685046 22837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:08.685139 22837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:08.685259 22837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:08.686527 22837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260806 02:55:08.686940 22837 leader_election.cc:302] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 863222b0fb1e4c3ba85d67313bd0bf56; no voters:
I20260806 02:55:08.687309 22837 leader_election.cc:288] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:08.687438 22844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:08.687747 22844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [term 1 LEADER]: Becoming Leader. State: Replica: 863222b0fb1e4c3ba85d67313bd0bf56, State: Running, Role: LEADER
I20260806 02:55:08.688146 22844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260806 02:55:08.688812 22837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:08.689411 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:08.674581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:39099 (local address 127.0.0.1:37859)
0806 02:55:08.675106 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:08.675121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:08.675158 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:55:08.675805 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:08.675823 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:08.676161 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:08.676508 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:08.677404 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:08.678701 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:08.679593 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:08.680145 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:08.680330 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:08.683474 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:08.683479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:08.683495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:08.683523 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:08.685584 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:08.686168 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:08.686173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:08.686177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:08.686309 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:08.686736 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:08.686741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:08.686746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:08.687236 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:08.687490 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:08.688177 (+ 687us) server_negotiation.cc:299] Negotiation successful
0806 02:55:08.688424 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":152,"threads_started":1}
I20260806 02:55:08.689954 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260806 02:55:08.690102 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:08.691337 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 863222b0fb1e4c3ba85d67313bd0bf56. Latest consensus state: current_term: 1 leader_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863222b0fb1e4c3ba85d67313bd0bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260806 02:55:08.691453 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:08.691932 22856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:08.693282 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:08.695314 22856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:08.701575 22856 catalog_manager.cc:1347] Generated new cluster ID: e5cea9e7591d4d8fa105607610cf8743
I20260806 02:55:08.701746 22856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:08.705840 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:08.695681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:51863 (local address 127.0.0.1:37859)
0806 02:55:08.695793 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:08.695797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:08.696567 (+ 770us) server_negotiation.cc:407] Connection header received
0806 02:55:08.696781 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:08.696784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:08.696834 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:08.696931 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:08.697744 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:08.698578 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:08.699725 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:08.700146 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:08.700278 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:08.702125 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:08.702130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:08.702133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:08.702160 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:08.703917 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:08.704433 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:08.704437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:08.704439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:08.704528 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:08.704881 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:08.704885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:08.704887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:08.705068 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:08.705229 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:08.705524 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 02:55:08.705625 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 02:55:08.716869 22856 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:08.717911 22856 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:08.732471 22856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 863222b0fb1e4c3ba85d67313bd0bf56: Generated new TSK 0
I20260806 02:55:08.733325 22856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:08.737941 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 22748
W20260806 02:55:08.752967 22729 connection.cc:535] client connection to 127.0.0.1:37859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37859): Connection reset by peer (error 104)
2026-08-06T02:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1653 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:08 dist-test-slave-3kk6 krb5kdc[22880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:08 dist-test-slave-3kk6 krb5kdc[22880](info): set up 2 sockets
Aug 06 02:55:08 dist-test-slave-3kk6 krb5kdc[22880](info): commencing operation
krb5kdc: starting...
W20260806 02:55:09.973277 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:10 dist-test-slave-3kk6 krb5kdc[22880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:10.116276 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:10.236760 22901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:10.237092 22901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:10.237138 22901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:10.241365 22901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:10.241465 22901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:10.241487 22901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:10.241504 22901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:10.241520 22901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:10.247390 22901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:10.248647 22901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:10.249809 22901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:10.259146 22908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:10.259131 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:10.259131 22907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:10.259712 22901 server_base.cc:1034] running on GCE node
I20260806 02:55:10.260314 22901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:10.261595 22901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:10.262746 22901 hybrid_clock.cc:648] HybridClock initialized: now 1785984910262714 us; error 48 us; skew 500 ppm
Aug 06 02:55:10 dist-test-slave-3kk6 krb5kdc[22880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:10.265928 22901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:10.267112 22901 webserver.cc:460] Webserver started at http://127.0.0.1:39283/ using document root <none> and password file <none>
I20260806 02:55:10.267714 22901 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:10.267779 22901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:10.268023 22901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:10.270228 22901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "595ef97543db40bb8dfdb7f4db944b6d"
format_stamp: "Formatted at 2026-08-06 02:55:10 on dist-test-slave-3kk6"
server_key: "54c1bf2ce2c0e7581b52be2bc8233138"
server_key_iv: "eedf93b8fd4c03953ea115a417836b1f"
server_key_version: "encryptionkey@0"
I20260806 02:55:10.270911 22901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "595ef97543db40bb8dfdb7f4db944b6d"
format_stamp: "Formatted at 2026-08-06 02:55:10 on dist-test-slave-3kk6"
server_key: "54c1bf2ce2c0e7581b52be2bc8233138"
server_key_iv: "eedf93b8fd4c03953ea115a417836b1f"
server_key_version: "encryptionkey@0"
I20260806 02:55:10.274978 22901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:10.277565 22921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:10.278966 22901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:10.279114 22901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "595ef97543db40bb8dfdb7f4db944b6d"
format_stamp: "Formatted at 2026-08-06 02:55:10 on dist-test-slave-3kk6"
server_key: "54c1bf2ce2c0e7581b52be2bc8233138"
server_key_iv: "eedf93b8fd4c03953ea115a417836b1f"
server_key_version: "encryptionkey@0"
I20260806 02:55:10.279237 22901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:10.299453 22901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:10.302855 22901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:10.303099 22901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:10.310974 22901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34931
I20260806 02:55:10.310984 22998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34931 every 8 connection(s)
I20260806 02:55:10.312079 22901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:10.314985 22999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:10.315399 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 22901
I20260806 02:55:10.315546 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:10.315845 25098 external_mini_cluster.cc:1428] Setting key 7eeb9506c8eacd7231789401e2091b12
I20260806 02:55:10.322975 22999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d: Bootstrap starting.
I20260806 02:55:10.326152 22999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:10.327260 22999 log.cc:824] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:10.327374 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:10.317424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:46369 (local address 127.0.0.1:34931)
0806 02:55:10.317888 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:10.317909 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:10.317956 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:55:10.318820 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:10.318846 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:10.319317 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:10.319702 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:10.320153 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:10.321722 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:10.322971 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:10.323692 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:10.323884 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:10.325092 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:10.325099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:10.325117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:10.325658 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:10.325742 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:10.325777 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:10.325949 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:10.326159 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 02:55:10.326316 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260806 02:55:10.329895 22999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d: No bootstrap required, opened a new log
I20260806 02:55:10.330935 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:10.334299 22999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595ef97543db40bb8dfdb7f4db944b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260806 02:55:10.334576 22999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:10.334637 22999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595ef97543db40bb8dfdb7f4db944b6d, State: Initialized, Role: FOLLOWER
I20260806 02:55:10.335179 22999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595ef97543db40bb8dfdb7f4db944b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260806 02:55:10.335374 22999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:10.335449 22999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:10.335554 22999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:10.336894 22999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595ef97543db40bb8dfdb7f4db944b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260806 02:55:10.337356 22999 leader_election.cc:302] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 595ef97543db40bb8dfdb7f4db944b6d; no voters:
I20260806 02:55:10.337800 22999 leader_election.cc:288] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:10.337947 23016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:10.338140 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:10.333149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:47325 (local address 127.0.0.1:34931)
0806 02:55:10.333299 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:10.333304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:10.333524 (+ 220us) server_negotiation.cc:407] Connection header received
0806 02:55:10.333649 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:10.333651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:10.333729 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:10.333808 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:10.334130 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:10.334815 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:10.336075 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:10.336325 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:10.336408 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:10.337486 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:10.337488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:10.337490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:10.337524 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:10.337562 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:10.337585 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:10.337669 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:10.337880 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 02:55:10.337923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:55:10.338305 23016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [term 1 LEADER]: Becoming Leader. State: Replica: 595ef97543db40bb8dfdb7f4db944b6d, State: Running, Role: LEADER
I20260806 02:55:10.338714 23016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595ef97543db40bb8dfdb7f4db944b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260806 02:55:10.339473 22999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:10.340097 23016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 595ef97543db40bb8dfdb7f4db944b6d. Latest consensus state: current_term: 1 leader_uuid: "595ef97543db40bb8dfdb7f4db944b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595ef97543db40bb8dfdb7f4db944b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260806 02:55:10.340343 23016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:10.340849 23029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:10.340945 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "595ef97543db40bb8dfdb7f4db944b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595ef97543db40bb8dfdb7f4db944b6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260806 02:55:10.341077 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:10.344161 23029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:10.351188 23029 catalog_manager.cc:1347] Generated new cluster ID: db3490b81bfc425eae61c3af19203698
I20260806 02:55:10.351303 23029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:10.359763 23029 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:10.360818 23029 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:10.367063 23029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 595ef97543db40bb8dfdb7f4db944b6d: Generated new TSK 0
I20260806 02:55:10.367839 23029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:10.374342 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 22901
W20260806 02:55:10.383308 22877 connection.cc:535] client connection to 127.0.0.1:34931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34931): Connection reset by peer (error 104)
2026-08-06T02:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:10 dist-test-slave-3kk6 krb5kdc[23044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:10 dist-test-slave-3kk6 krb5kdc[23044](info): set up 2 sockets
Aug 06 02:55:10 dist-test-slave-3kk6 krb5kdc[23044](info): commencing operation
krb5kdc: starting...
W20260806 02:55:11.604362 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:11 dist-test-slave-3kk6 krb5kdc[23044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:55:11.742167 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:11.860062 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:11.860424 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:11.860504 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:11.864602 23060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:11.864724 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:11.864765 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:11.864794 23060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:11.864820 23060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:11.870455 23060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:11.871697 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:11.872777 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:11.881775 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:11.881839 23069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:11.881855 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:11.882009 23060 server_base.cc:1034] running on GCE node
I20260806 02:55:11.882766 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:11.883849 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:11.885037 23060 hybrid_clock.cc:648] HybridClock initialized: now 1785984911885025 us; error 26 us; skew 500 ppm
Aug 06 02:55:11 dist-test-slave-3kk6 krb5kdc[23044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:11.888021 23060 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:55:11.889021 23060 webserver.cc:460] Webserver started at http://127.0.0.1:40361/ using document root <none> and password file <none>
I20260806 02:55:11.889612 23060 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:11.889678 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:11.889923 23060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:11.892248 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "905a1fe09b9c42d0a0f21abc32cad1a8"
format_stamp: "Formatted at 2026-08-06 02:55:11 on dist-test-slave-3kk6"
server_key: "dafda9f7e5d622970f244de1ba173a74"
server_key_iv: "72200f848a9d678a0397111c9030e32a"
server_key_version: "encryptionkey@0"
I20260806 02:55:11.892902 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "905a1fe09b9c42d0a0f21abc32cad1a8"
format_stamp: "Formatted at 2026-08-06 02:55:11 on dist-test-slave-3kk6"
server_key: "dafda9f7e5d622970f244de1ba173a74"
server_key_iv: "72200f848a9d678a0397111c9030e32a"
server_key_version: "encryptionkey@0"
I20260806 02:55:11.897085 23060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:55:11.899802 23084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:11.901082 23060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:11.901250 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "905a1fe09b9c42d0a0f21abc32cad1a8"
format_stamp: "Formatted at 2026-08-06 02:55:11 on dist-test-slave-3kk6"
server_key: "dafda9f7e5d622970f244de1ba173a74"
server_key_iv: "72200f848a9d678a0397111c9030e32a"
server_key_version: "encryptionkey@0"
I20260806 02:55:11.901386 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:11.906289 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:11.909621 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:11.909852 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:11.917557 23060 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45441
I20260806 02:55:11.917588 23165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45441 every 8 connection(s)
I20260806 02:55:11.918557 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:11.920738 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 23060
I20260806 02:55:11.920892 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:11.921192 25098 external_mini_cluster.cc:1428] Setting key f0d783ddcffc08bd250e67cb903d105e
I20260806 02:55:11.921598 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:11.927278 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:11.922780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:58033 (local address 127.0.0.1:45441)
0806 02:55:11.923233 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:11.923264 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:11.923309 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:55:11.924116 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:11.924140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:11.924519 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:11.924931 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:11.925741 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:11.925747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:11.925764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:11.925778 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:11.925852 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:11.925886 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:11.926047 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:11.926117 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 02:55:11.926281 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260806 02:55:11.929656 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8: Bootstrap starting.
I20260806 02:55:11.930809 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:11.932466 23166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:11.933477 23166 log.cc:824] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:11.934976 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:11.933079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:37367 (local address 127.0.0.1:45441)
0806 02:55:11.933208 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:11.933213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:11.933392 (+ 179us) server_negotiation.cc:407] Connection header received
0806 02:55:11.933566 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:11.933569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:11.933614 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:11.933698 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:11.934524 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:11.934527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:11.934531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:11.934543 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:11.934574 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:11.934590 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:11.934703 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:11.934806 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 02:55:11.934853 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:55:11.935849 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8: No bootstrap required, opened a new log
I20260806 02:55:11.939670 23166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260806 02:55:11.939939 23166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:11.940034 23166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905a1fe09b9c42d0a0f21abc32cad1a8, State: Initialized, Role: FOLLOWER
I20260806 02:55:11.940548 23166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260806 02:55:11.940709 23166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:11.940789 23166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:11.940940 23166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:11.942238 23166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260806 02:55:11.942721 23166 leader_election.cc:302] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 905a1fe09b9c42d0a0f21abc32cad1a8; no voters:
I20260806 02:55:11.943163 23166 leader_election.cc:288] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:11.943298 23178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:11.943600 23178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [term 1 LEADER]: Becoming Leader. State: Replica: 905a1fe09b9c42d0a0f21abc32cad1a8, State: Running, Role: LEADER
I20260806 02:55:11.943986 23178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } }
I20260806 02:55:11.944612 23166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:11.945869 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } } }
I20260806 02:55:11.945921 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905a1fe09b9c42d0a0f21abc32cad1a8. Latest consensus state: current_term: 1 leader_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905a1fe09b9c42d0a0f21abc32cad1a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45441 } } }
I20260806 02:55:11.946015 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:11.946036 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:11.946571 23189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:11.950060 23189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:11.957036 23189 catalog_manager.cc:1347] Generated new cluster ID: 5ddcf6ea89dc414f91ae09e73b232e8e
I20260806 02:55:11.957145 23189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:11.968633 23189 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:11.969014 23189 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:11.979318 23189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 905a1fe09b9c42d0a0f21abc32cad1a8: Generated new TSK 0
I20260806 02:55:11.980094 23189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:11.989249 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 23060
2026-08-06T02:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1620 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:12 dist-test-slave-3kk6 krb5kdc[23203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:12 dist-test-slave-3kk6 krb5kdc[23203](info): set up 2 sockets
Aug 06 02:55:12 dist-test-slave-3kk6 krb5kdc[23203](info): commencing operation
krb5kdc: starting...
W20260806 02:55:13.230803 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:13 dist-test-slave-3kk6 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:13.375057 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:42015
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:13.496860 23224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:13.497226 23224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:13.497308 23224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:13.501595 23224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:13.501720 23224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:13.501761 23224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:13.501848 23224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:13.501888 23224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:13.507671 23224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:42015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:13.508908 23224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:13.510007 23224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:13.519058 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:13.519286 23231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:13.519075 23230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:13.519544 23224 server_base.cc:1034] running on GCE node
I20260806 02:55:13.520211 23224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:13.521538 23224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:13.522816 23224 hybrid_clock.cc:648] HybridClock initialized: now 1785984913522780 us; error 70 us; skew 500 ppm
Aug 06 02:55:13 dist-test-slave-3kk6 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984913, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:13.529083 23224 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:13.530223 23224 webserver.cc:460] Webserver started at http://127.24.130.190:36431/ using document root <none> and password file <none>
I20260806 02:55:13.530938 23224 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:13.531047 23224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:13.531322 23224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:13.533651 23224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "2102b29e064646968f7f87fe2dc276b2"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-3kk6"
server_key: "3123522ef836fcb6ce02a8de32234cfb"
server_key_iv: "5e6797f4204d1eae379b03410c19655f"
server_key_version: "encryptionkey@0"
I20260806 02:55:13.534301 23224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "2102b29e064646968f7f87fe2dc276b2"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-3kk6"
server_key: "3123522ef836fcb6ce02a8de32234cfb"
server_key_iv: "5e6797f4204d1eae379b03410c19655f"
server_key_version: "encryptionkey@0"
I20260806 02:55:13.539007 23224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260806 02:55:13.542140 23246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:13.543723 23224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:13.543910 23224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "2102b29e064646968f7f87fe2dc276b2"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-3kk6"
server_key: "3123522ef836fcb6ce02a8de32234cfb"
server_key_iv: "5e6797f4204d1eae379b03410c19655f"
server_key_version: "encryptionkey@0"
I20260806 02:55:13.544064 23224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:13.560840 23224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:13.564183 23224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:13.564419 23224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:13.572332 23224 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:42015
I20260806 02:55:13.572360 23321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:42015 every 8 connection(s)
I20260806 02:55:13.573395 23224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:13.573875 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 23224
I20260806 02:55:13.574054 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:13.574347 25098 external_mini_cluster.cc:1428] Setting key 1b097804d21cd69ce42882f4180966d1
I20260806 02:55:13.577463 23322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:55:13 dist-test-slave-3kk6 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:13.583670 23322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2: Bootstrap starting.
I20260806 02:55:13.585996 23322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:13.586845 23322 log.cc:824] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:13.589344 23322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2: No bootstrap required, opened a new log
I20260806 02:55:13.589748 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:13.575709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51441 (local address 127.24.130.190:42015)
0806 02:55:13.576131 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:13.576146 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:13.576185 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:55:13.576852 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:13.576869 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:13.577218 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:13.577587 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:13.578491 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:13.579608 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:13.580549 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:13.581044 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:13.581225 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:13.584302 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:13.584308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:13.584321 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:13.584352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:13.586237 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:13.586826 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:13.586838 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:13.586843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:13.586984 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:13.587411 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:13.587414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:13.587415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:13.587931 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:13.588170 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:13.588491 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 02:55:13.588740 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260806 02:55:13.593192 23322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2102b29e064646968f7f87fe2dc276b2" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 42015 } }
I20260806 02:55:13.593497 23322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:13.593612 23322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2102b29e064646968f7f87fe2dc276b2, State: Initialized, Role: FOLLOWER
I20260806 02:55:13.593674 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:13.594110 23322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2102b29e064646968f7f87fe2dc276b2" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 42015 } }
I20260806 02:55:13.594288 23322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:13.594384 23322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:13.594513 23322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:13.595845 23322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2102b29e064646968f7f87fe2dc276b2" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 42015 } }
I20260806 02:55:13.596314 23322 leader_election.cc:302] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2102b29e064646968f7f87fe2dc276b2; no voters:
I20260806 02:55:13.596724 23322 leader_election.cc:288] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:13.597007 23339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:13.597296 23339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [term 1 LEADER]: Becoming Leader. State: Replica: 2102b29e064646968f7f87fe2dc276b2, State: Running, Role: LEADER
I20260806 02:55:13.597700 23339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2102b29e064646968f7f87fe2dc276b2" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 42015 } }
I20260806 02:55:13.598274 23322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:13.599470 23342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2102b29e064646968f7f87fe2dc276b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2102b29e064646968f7f87fe2dc276b2" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 42015 } } }
I20260806 02:55:13.599678 23342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:13.600008 23342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2102b29e064646968f7f87fe2dc276b2. Latest consensus state: current_term: 1 leader_uuid: "2102b29e064646968f7f87fe2dc276b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2102b29e064646968f7f87fe2dc276b2" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 42015 } } }
I20260806 02:55:13.600159 23342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:13.600502 23353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:13.604897 23353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:13.606720 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:13.596192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57503 (local address 127.24.130.190:42015)
0806 02:55:13.596344 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:13.596348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:13.596683 (+ 335us) server_negotiation.cc:407] Connection header received
0806 02:55:13.596891 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:13.596894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:13.596950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:13.597037 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:13.597972 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:13.598795 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:13.600104 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:13.600483 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:13.600594 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:13.602477 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:13.602480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:13.602483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:13.602512 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:13.604392 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:13.605101 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:13.605105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:13.605106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:13.605209 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:13.605633 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:13.605637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:13.605639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:13.605900 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:13.606065 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:13.606405 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:55:13.606523 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:55:13.612351 23353 catalog_manager.cc:1347] Generated new cluster ID: 32edf05b2f5a4ea5bef2a9af69b2ac76
I20260806 02:55:13.612447 23353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:13.631443 23353 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:13.632871 23353 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:13.639837 23353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2102b29e064646968f7f87fe2dc276b2: Generated new TSK 0
I20260806 02:55:13.640692 23353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:13.647140 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 23224
W20260806 02:55:13.662322 23201 connection.cc:535] client connection to 127.24.130.190:42015 recv error: Network error: failed to read from TLS socket (remote: 127.24.130.190:42015): Connection reset by peer (error 104)
2026-08-06T02:55:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1658 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:13 dist-test-slave-3kk6 krb5kdc[23366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:13 dist-test-slave-3kk6 krb5kdc[23366](info): set up 2 sockets
Aug 06 02:55:13 dist-test-slave-3kk6 krb5kdc[23366](info): commencing operation
krb5kdc: starting...
W20260806 02:55:14.909349 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:15 dist-test-slave-3kk6 krb5kdc[23366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:15.054342 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:34711
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:34711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:15.178126 23394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:15.178506 23394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:15.178586 23394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:15.182981 23394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:15.183092 23394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:15.183130 23394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:15.183172 23394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:15.183209 23394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:15.189278 23394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:34711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:34711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:15.190567 23394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:15.191820 23394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:15.201484 23399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:15.201588 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:15.201658 23394 server_base.cc:1034] running on GCE node
W20260806 02:55:15.201484 23405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:15.202643 23394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:15.203999 23394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:15.205221 23394 hybrid_clock.cc:648] HybridClock initialized: now 1785984915205210 us; error 49 us; skew 500 ppm
Aug 06 02:55:15 dist-test-slave-3kk6 krb5kdc[23366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984915, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:15.208420 23394 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:15.209424 23394 webserver.cc:460] Webserver started at http://127.24.130.190:35407/ using document root <none> and password file <none>
I20260806 02:55:15.210029 23394 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:15.210130 23394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:15.210392 23394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:15.212805 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "9e718499528c47cbaf290a7379fcfd6a"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-3kk6"
server_key: "9154c8747283df01195d2a899612d438"
server_key_iv: "290c202916705b8d6125c635a412d667"
server_key_version: "encryptionkey@0"
I20260806 02:55:15.213434 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "9e718499528c47cbaf290a7379fcfd6a"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-3kk6"
server_key: "9154c8747283df01195d2a899612d438"
server_key_iv: "290c202916705b8d6125c635a412d667"
server_key_version: "encryptionkey@0"
I20260806 02:55:15.217607 23394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:55:15.220639 23413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:15.222026 23394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:55:15.222204 23394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "9e718499528c47cbaf290a7379fcfd6a"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-3kk6"
server_key: "9154c8747283df01195d2a899612d438"
server_key_iv: "290c202916705b8d6125c635a412d667"
server_key_version: "encryptionkey@0"
I20260806 02:55:15.222352 23394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:15.237181 23394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:15.240896 23394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:15.241161 23394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:15.249024 23394 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:34711
I20260806 02:55:15.249053 23496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:34711 every 8 connection(s)
I20260806 02:55:15.250128 23394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:15.253159 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:15.253541 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 23394
I20260806 02:55:15.253698 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:15.254025 25098 external_mini_cluster.cc:1428] Setting key bb7ee25e58a9f52b337700a3bc38fe12
I20260806 02:55:15.261047 23497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a: Bootstrap starting.
I20260806 02:55:15.263823 23497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:15.264173 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:15.255703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47737 (local address 127.24.130.190:34711)
0806 02:55:15.256174 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:15.256190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:15.256228 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:55:15.256932 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:15.256951 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:15.257315 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:15.257600 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:15.258017 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:15.259179 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:15.260442 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:15.260953 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:15.261090 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:15.262105 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:15.262110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:15.262123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:15.262549 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:15.262609 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:15.262633 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:15.262891 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:15.263138 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 02:55:15.263315 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20260806 02:55:15.264678 23497 log.cc:824] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:15.267274 23497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a: No bootstrap required, opened a new log
I20260806 02:55:15.267459 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:15.271946 23497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e718499528c47cbaf290a7379fcfd6a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34711 } }
I20260806 02:55:15.272269 23497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:15.272363 23497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e718499528c47cbaf290a7379fcfd6a, State: Initialized, Role: FOLLOWER
I20260806 02:55:15.272904 23497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e718499528c47cbaf290a7379fcfd6a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34711 } }
I20260806 02:55:15.273084 23497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:15.273180 23497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:15.273299 23497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:15.274288 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:15.269729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33049 (local address 127.24.130.190:34711)
0806 02:55:15.269893 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:15.269896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:15.270142 (+ 246us) server_negotiation.cc:407] Connection header received
0806 02:55:15.270227 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:15.270231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:15.270301 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:15.270396 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:15.270645 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:15.271483 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:15.272407 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:15.272768 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:15.272878 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:15.273844 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:15.273848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:15.273851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:15.273865 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:15.273896 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:15.273909 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:15.274004 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:15.274107 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 02:55:15.274145 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:55:15.274813 23497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e718499528c47cbaf290a7379fcfd6a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34711 } }
I20260806 02:55:15.275218 23497 leader_election.cc:302] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e718499528c47cbaf290a7379fcfd6a; no voters:
I20260806 02:55:15.275658 23497 leader_election.cc:288] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:15.275812 23513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:15.276075 23513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [term 1 LEADER]: Becoming Leader. State: Replica: 9e718499528c47cbaf290a7379fcfd6a, State: Running, Role: LEADER
I20260806 02:55:15.276409 23513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e718499528c47cbaf290a7379fcfd6a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34711 } }
I20260806 02:55:15.277110 23497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:15.278345 23514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e718499528c47cbaf290a7379fcfd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e718499528c47cbaf290a7379fcfd6a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34711 } } }
I20260806 02:55:15.278491 23514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:15.278934 23525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:15.279369 23515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e718499528c47cbaf290a7379fcfd6a. Latest consensus state: current_term: 1 leader_uuid: "9e718499528c47cbaf290a7379fcfd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e718499528c47cbaf290a7379fcfd6a" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 34711 } } }
I20260806 02:55:15.279542 23515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:15.282160 23525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:15.288501 23525 catalog_manager.cc:1347] Generated new cluster ID: 1f20199f2d1e4191ad7f9b6a4988dc9b
I20260806 02:55:15.288607 23525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:15.307595 23525 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:15.309033 23525 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:15.324899 23525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e718499528c47cbaf290a7379fcfd6a: Generated new TSK 0
I20260806 02:55:15.325862 23525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:15.338753 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 23394
W20260806 02:55:15.352137 23364 connection.cc:535] client connection to 127.24.130.190:34711 recv error: Network error: failed to read from TLS socket (remote: 127.24.130.190:34711): Connection reset by peer (error 104)
2026-08-06T02:55:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1689 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:15 dist-test-slave-3kk6 krb5kdc[23541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:15 dist-test-slave-3kk6 krb5kdc[23541](info): set up 2 sockets
Aug 06 02:55:15 dist-test-slave-3kk6 krb5kdc[23541](info): commencing operation
krb5kdc: starting...
W20260806 02:55:16.565446 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:16 dist-test-slave-3kk6 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:16.702414 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39577
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:16.821369 23556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:16.821719 23556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:16.821815 23556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:16.825978 23556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:16.826098 23556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:16.826141 23556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:16.826201 23556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:16.826237 23556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:16.831825 23556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:39577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:16.833047 23556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:16.834216 23556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:16.843298 23567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:16.843389 23564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:16.843304 23563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:16.843484 23556 server_base.cc:1034] running on GCE node
I20260806 02:55:16.844247 23556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:16.845778 23556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:16.846987 23556 hybrid_clock.cc:648] HybridClock initialized: now 1785984916846962 us; error 49 us; skew 500 ppm
Aug 06 02:55:16 dist-test-slave-3kk6 krb5kdc[23541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984916, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:16.850214 23556 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:16.851238 23556 webserver.cc:460] Webserver started at http://127.24.130.190:41337/ using document root <none> and password file <none>
I20260806 02:55:16.851804 23556 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:16.851869 23556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:16.852049 23556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:16.854200 23556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "7ed0413d36d54f8cafdae55158e3bb98"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-3kk6"
server_key: "4d3cf4553dd5d25fc8a67f81a6a45729"
server_key_iv: "f23247b52abf33901b31c6f5e37fe382"
server_key_version: "encryptionkey@0"
I20260806 02:55:16.854841 23556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "7ed0413d36d54f8cafdae55158e3bb98"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-3kk6"
server_key: "4d3cf4553dd5d25fc8a67f81a6a45729"
server_key_iv: "f23247b52abf33901b31c6f5e37fe382"
server_key_version: "encryptionkey@0"
I20260806 02:55:16.858841 23556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:16.861361 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:16.862615 23556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:55:16.862831 23556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "7ed0413d36d54f8cafdae55158e3bb98"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-3kk6"
server_key: "4d3cf4553dd5d25fc8a67f81a6a45729"
server_key_iv: "f23247b52abf33901b31c6f5e37fe382"
server_key_version: "encryptionkey@0"
I20260806 02:55:16.862951 23556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:16.867774 23556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:16.870955 23556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:16.871170 23556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:16.878520 23556 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:39577
I20260806 02:55:16.878577 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:39577 every 8 connection(s)
I20260806 02:55:16.879563 23556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:16.880541 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 23556
I20260806 02:55:16.880715 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:16.881026 25098 external_mini_cluster.cc:1428] Setting key 6716de7f17fff875e28c55ab8c8e7d03
I20260806 02:55:16.883173 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:16.886579 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.882483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41129 (local address 127.24.130.190:39577)
0806 02:55:16.882957 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:16.882983 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:16.883021 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:55:16.883734 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:16.883752 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:16.884035 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:16.884327 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:16.885300 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:16.885305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:16.885318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:16.885328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:16.885384 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:16.885411 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:16.885527 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:16.885655 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 02:55:16.885828 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":197,"threads_started":1}
I20260806 02:55:16.889545 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:16.889598 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98: Bootstrap starting.
I20260806 02:55:16.892648 23649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:16.893556 23649 log.cc:824] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:16.893811 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.892035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39647 (local address 127.24.130.190:39577)
0806 02:55:16.892182 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:16.892187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:16.892357 (+ 170us) server_negotiation.cc:407] Connection header received
0806 02:55:16.892454 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:16.892459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:16.892506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:16.892609 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:16.893423 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:16.893428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:16.893432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:16.893446 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:16.893476 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:16.893488 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:16.893590 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:16.893655 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:55:16.893697 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:55:16.895735 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98: No bootstrap required, opened a new log
I20260806 02:55:16.899497 23649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed0413d36d54f8cafdae55158e3bb98" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39577 } }
I20260806 02:55:16.899792 23649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:16.899847 23649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed0413d36d54f8cafdae55158e3bb98, State: Initialized, Role: FOLLOWER
I20260806 02:55:16.900317 23649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed0413d36d54f8cafdae55158e3bb98" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39577 } }
I20260806 02:55:16.900511 23649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:16.900591 23649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:16.900738 23649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:16.901844 23649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed0413d36d54f8cafdae55158e3bb98" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39577 } }
I20260806 02:55:16.902174 23649 leader_election.cc:302] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ed0413d36d54f8cafdae55158e3bb98; no voters:
I20260806 02:55:16.902511 23649 leader_election.cc:288] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:16.902630 23664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:16.902943 23664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [term 1 LEADER]: Becoming Leader. State: Replica: 7ed0413d36d54f8cafdae55158e3bb98, State: Running, Role: LEADER
I20260806 02:55:16.903334 23664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed0413d36d54f8cafdae55158e3bb98" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39577 } }
I20260806 02:55:16.903959 23649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:16.905103 23665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ed0413d36d54f8cafdae55158e3bb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed0413d36d54f8cafdae55158e3bb98" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39577 } } }
I20260806 02:55:16.905299 23665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:16.905812 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ed0413d36d54f8cafdae55158e3bb98. Latest consensus state: current_term: 1 leader_uuid: "7ed0413d36d54f8cafdae55158e3bb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed0413d36d54f8cafdae55158e3bb98" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 39577 } } }
I20260806 02:55:16.905929 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:16.905812 23673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:16.909418 23673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:16.915405 23673 catalog_manager.cc:1347] Generated new cluster ID: 5b615fbd82fa41dfa25495378f7f8931
I20260806 02:55:16.915503 23673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:16.932646 23673 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:16.933127 23673 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:16.946565 23673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ed0413d36d54f8cafdae55158e3bb98: Generated new TSK 0
I20260806 02:55:16.947470 23673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:16.961485 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 23556
2026-08-06T02:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:17 dist-test-slave-3kk6 krb5kdc[23693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:17 dist-test-slave-3kk6 krb5kdc[23693](info): set up 2 sockets
Aug 06 02:55:17 dist-test-slave-3kk6 krb5kdc[23693](info): commencing operation
krb5kdc: starting...
W20260806 02:55:18.185405 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:18 dist-test-slave-3kk6 krb5kdc[23693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:18.323263 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41867
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:18.441656 23710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:18.442026 23710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:18.442108 23710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:18.446328 23710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:18.446468 23710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:18.446506 23710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:18.446547 23710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:18.446580 23710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:18.452226 23710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:18.453465 23710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:18.454586 23710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:18.463451 23718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:18.463451 23717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:18.463465 23720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:18.463589 23710 server_base.cc:1034] running on GCE node
I20260806 02:55:18.464355 23710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:18.465497 23710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:18.466714 23710 hybrid_clock.cc:648] HybridClock initialized: now 1785984918466688 us; error 34 us; skew 500 ppm
Aug 06 02:55:18 dist-test-slave-3kk6 krb5kdc[23693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984918, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:18.469664 23710 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:18.470588 23710 webserver.cc:460] Webserver started at http://127.24.130.190:33679/ using document root <none> and password file <none>
I20260806 02:55:18.471179 23710 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:18.471246 23710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:18.471496 23710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:18.473615 23710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "74f99cd815064639a6b9c8354333d163"
format_stamp: "Formatted at 2026-08-06 02:55:18 on dist-test-slave-3kk6"
server_key: "88278f3c529465bd1e5dddc8eb8e846d"
server_key_iv: "fe9a6a17458ac9568d7e20f7e8e16a47"
server_key_version: "encryptionkey@0"
I20260806 02:55:18.474210 23710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "74f99cd815064639a6b9c8354333d163"
format_stamp: "Formatted at 2026-08-06 02:55:18 on dist-test-slave-3kk6"
server_key: "88278f3c529465bd1e5dddc8eb8e846d"
server_key_iv: "fe9a6a17458ac9568d7e20f7e8e16a47"
server_key_version: "encryptionkey@0"
I20260806 02:55:18.478122 23710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:55:18.480636 23733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:18.481876 23710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:55:18.482013 23710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "74f99cd815064639a6b9c8354333d163"
format_stamp: "Formatted at 2026-08-06 02:55:18 on dist-test-slave-3kk6"
server_key: "88278f3c529465bd1e5dddc8eb8e846d"
server_key_iv: "fe9a6a17458ac9568d7e20f7e8e16a47"
server_key_version: "encryptionkey@0"
I20260806 02:55:18.482131 23710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:18.522275 23710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:18.525981 23710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:18.526221 23710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:18.534340 23710 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:41867
I20260806 02:55:18.534368 23811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:41867 every 8 connection(s)
I20260806 02:55:18.535413 23710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:18.538276 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:18.542237 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 23710
I20260806 02:55:18.542415 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:18.542758 25098 external_mini_cluster.cc:1428] Setting key a20da51678be4f973477f7e2c1a4ae47
I20260806 02:55:18.545181 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163: Bootstrap starting.
I20260806 02:55:18.547591 23812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:18.548394 23812 log.cc:824] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:18.550525 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163: No bootstrap required, opened a new log
I20260806 02:55:18.554258 23812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f99cd815064639a6b9c8354333d163" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41867 } }
I20260806 02:55:18.554563 23812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:18.554654 23812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f99cd815064639a6b9c8354333d163, State: Initialized, Role: FOLLOWER
I20260806 02:55:18.555197 23812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f99cd815064639a6b9c8354333d163" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41867 } }
I20260806 02:55:18.555371 23812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:18.555481 23812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:18.555595 23812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:18.556636 23812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f99cd815064639a6b9c8354333d163" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41867 } }
I20260806 02:55:18.557000 23812 leader_election.cc:302] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f99cd815064639a6b9c8354333d163; no voters:
I20260806 02:55:18.557389 23812 leader_election.cc:288] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:18.557529 23818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 1 FOLLOWER]: Leader election won for term 1
Aug 06 02:55:18 dist-test-slave-3kk6 krb5kdc[23693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:18.557771 23818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [term 1 LEADER]: Becoming Leader. State: Replica: 74f99cd815064639a6b9c8354333d163, State: Running, Role: LEADER
I20260806 02:55:18.558135 23818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f99cd815064639a6b9c8354333d163" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41867 } }
I20260806 02:55:18.558801 23812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:18.559506 23821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f99cd815064639a6b9c8354333d163. Latest consensus state: current_term: 1 leader_uuid: "74f99cd815064639a6b9c8354333d163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f99cd815064639a6b9c8354333d163" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41867 } } }
I20260806 02:55:18.559636 23821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:18.560415 23820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f99cd815064639a6b9c8354333d163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f99cd815064639a6b9c8354333d163" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 41867 } } }
I20260806 02:55:18.560585 23820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:18.560920 23827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:18.564388 23827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:18.564659 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:18.544427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36153 (local address 127.24.130.190:41867)
0806 02:55:18.544940 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:18.544958 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:18.544996 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:55:18.545756 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:18.545775 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:18.546141 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:18.546496 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:18.547427 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.548572 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.549691 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.550188 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.550368 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:18.558547 (+ 8179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:18.558554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:18.558573 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:18.558613 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:18.561073 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:18.561648 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:18.561657 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:18.561667 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:18.561819 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:18.562248 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:18.562252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:18.562254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:18.562839 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:18.563080 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:18.563375 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 02:55:18.563694 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":150,"threads_started":1}
I20260806 02:55:18.568120 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:18.570569 23827 catalog_manager.cc:1347] Generated new cluster ID: a03e3989ae154bb088a6aadb48dd8932
I20260806 02:55:18.570672 23827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:18.579066 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:18.570764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37157 (local address 127.24.130.190:41867)
0806 02:55:18.570883 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:18.570888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:18.571391 (+ 503us) server_negotiation.cc:407] Connection header received
0806 02:55:18.571558 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:18.571561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:18.571614 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:18.571716 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:18.572418 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.572978 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.573819 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.574077 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.574199 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:18.575711 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:18.575714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:18.575717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:18.575742 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:18.577285 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:18.577746 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:18.577749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:18.577751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:18.577837 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:18.578133 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:18.578137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:18.578139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:18.578328 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:18.578469 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:18.578759 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:55:18.578884 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:55:18.596540 23827 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:18.597533 23827 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:18.605597 23827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f99cd815064639a6b9c8354333d163: Generated new TSK 0
I20260806 02:55:18.606329 23827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:18.611263 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 23710
2026-08-06T02:55:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1652 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:18 dist-test-slave-3kk6 krb5kdc[23856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:18 dist-test-slave-3kk6 krb5kdc[23856](info): set up 2 sockets
Aug 06 02:55:18 dist-test-slave-3kk6 krb5kdc[23856](info): commencing operation
krb5kdc: starting...
W20260806 02:55:19.838056 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:19 dist-test-slave-3kk6 krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:19.980079 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:37199
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:20.106089 23874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:20.106418 23874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:20.106464 23874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:20.110821 23874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:20.110911 23874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:20.110931 23874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:20.110949 23874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:20.110965 23874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:20.116835 23874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:20.118022 23874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:20.119207 23874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:20.128734 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:20.128734 23879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:20.128734 23886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:20.129304 23874 server_base.cc:1034] running on GCE node
I20260806 02:55:20.129998 23874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:20.131338 23874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:20.132508 23874 hybrid_clock.cc:648] HybridClock initialized: now 1785984920132492 us; error 39 us; skew 500 ppm
Aug 06 02:55:20 dist-test-slave-3kk6 krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984920, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:20.135715 23874 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:20.136670 23874 webserver.cc:460] Webserver started at http://127.24.130.190:36043/ using document root <none> and password file <none>
I20260806 02:55:20.137226 23874 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:20.137300 23874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:20.137481 23874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:20.139707 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "2747424ebeab4a8aa4b6183183345829"
format_stamp: "Formatted at 2026-08-06 02:55:20 on dist-test-slave-3kk6"
server_key: "4bb08029ef5992884082485190290ea3"
server_key_iv: "299b49e19aca30d86420562686dafdcf"
server_key_version: "encryptionkey@0"
I20260806 02:55:20.140353 23874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "2747424ebeab4a8aa4b6183183345829"
format_stamp: "Formatted at 2026-08-06 02:55:20 on dist-test-slave-3kk6"
server_key: "4bb08029ef5992884082485190290ea3"
server_key_iv: "299b49e19aca30d86420562686dafdcf"
server_key_version: "encryptionkey@0"
I20260806 02:55:20.144444 23874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:55:20.146947 23896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:20.148224 23874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:55:20.148384 23874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "2747424ebeab4a8aa4b6183183345829"
format_stamp: "Formatted at 2026-08-06 02:55:20 on dist-test-slave-3kk6"
server_key: "4bb08029ef5992884082485190290ea3"
server_key_iv: "299b49e19aca30d86420562686dafdcf"
server_key_version: "encryptionkey@0"
I20260806 02:55:20.148507 23874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:20.160192 23874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:20.164080 23874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:20.164312 23874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:20.171711 23874 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:37199
I20260806 02:55:20.171809 23967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:37199 every 8 connection(s)
I20260806 02:55:20.172802 23874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:20.175884 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:20.178809 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 23874
I20260806 02:55:20.178978 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:20.179241 25098 external_mini_cluster.cc:1428] Setting key 619aaa03c573b8a26aa8627bba032489
I20260806 02:55:20.182273 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829: Bootstrap starting.
I20260806 02:55:20.184679 23968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:20.185468 23968 log.cc:824] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:20.187542 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829: No bootstrap required, opened a new log
I20260806 02:55:20.189070 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.180813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51597 (local address 127.24.130.190:37199)
0806 02:55:20.181386 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.181403 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.181444 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:55:20.182145 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.182171 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.182618 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.183076 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.183516 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.184520 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.185539 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.186084 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.186282 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.187280 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.187286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.187303 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:20.187764 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.187825 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:20.187850 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.187965 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.188116 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.188344 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":188,"threads_started":1}
I20260806 02:55:20.191850 23968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747424ebeab4a8aa4b6183183345829" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37199 } }
I20260806 02:55:20.192163 23968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:20.192205 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:20.192247 23968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2747424ebeab4a8aa4b6183183345829, State: Initialized, Role: FOLLOWER
I20260806 02:55:20.192679 23968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747424ebeab4a8aa4b6183183345829" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37199 } }
I20260806 02:55:20.192842 23968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:20.192937 23968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:20.193060 23968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:20.194180 23968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747424ebeab4a8aa4b6183183345829" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37199 } }
I20260806 02:55:20.194573 23968 leader_election.cc:302] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2747424ebeab4a8aa4b6183183345829; no voters:
I20260806 02:55:20.195009 23968 leader_election.cc:288] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:20.195079 23982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:20.195328 23982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [term 1 LEADER]: Becoming Leader. State: Replica: 2747424ebeab4a8aa4b6183183345829, State: Running, Role: LEADER
I20260806 02:55:20.195706 23982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747424ebeab4a8aa4b6183183345829" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37199 } }
I20260806 02:55:20.196446 23968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:20.197556 23983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2747424ebeab4a8aa4b6183183345829. Latest consensus state: current_term: 1 leader_uuid: "2747424ebeab4a8aa4b6183183345829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747424ebeab4a8aa4b6183183345829" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37199 } } }
I20260806 02:55:20.197701 23983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:20.198006 23984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2747424ebeab4a8aa4b6183183345829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747424ebeab4a8aa4b6183183345829" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 37199 } } }
I20260806 02:55:20.198093 23984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:20.198127 23991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:20.199787 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.194589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48901 (local address 127.24.130.190:37199)
0806 02:55:20.194774 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.194779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.195050 (+ 271us) server_negotiation.cc:407] Connection header received
0806 02:55:20.195225 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.195228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.195283 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.195358 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.195822 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.196373 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.197584 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.197963 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.198056 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.199290 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.199294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.199297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:20.199314 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.199344 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:20.199358 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.199449 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.199582 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.199647 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:55:20.202024 23991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:20.208559 23991 catalog_manager.cc:1347] Generated new cluster ID: 317fadfc11aa47fdabb9b194c45e97a8
I20260806 02:55:20.208683 23991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:20.221642 23991 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:20.223078 23991 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:20.234719 23991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2747424ebeab4a8aa4b6183183345829: Generated new TSK 0
I20260806 02:55:20.235736 23991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:20.243119 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 23874
2026-08-06T02:55:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1627 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:20 dist-test-slave-3kk6 krb5kdc[24008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:20 dist-test-slave-3kk6 krb5kdc[24008](info): set up 2 sockets
Aug 06 02:55:20 dist-test-slave-3kk6 krb5kdc[24008](info): commencing operation
krb5kdc: starting...
W20260806 02:55:21.486219 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:21 dist-test-slave-3kk6 krb5kdc[24008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:21.625995 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:32973
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:32973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:21.743191 24026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:21.743566 24026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:21.743655 24026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:21.747912 24026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:21.748027 24026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:21.748067 24026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:21.748129 24026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:21.748170 24026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:21.753840 24026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:32973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:32973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:21.755184 24026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:21.756294 24026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:21.765364 24035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:21.765432 24031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:21.765622 24026 server_base.cc:1034] running on GCE node
W20260806 02:55:21.765734 24032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:21.766369 24026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:21.767647 24026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:21.768855 24026 hybrid_clock.cc:648] HybridClock initialized: now 1785984921768842 us; error 42 us; skew 500 ppm
Aug 06 02:55:21 dist-test-slave-3kk6 krb5kdc[24008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984921, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:21.771945 24026 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:21.772933 24026 webserver.cc:460] Webserver started at http://127.24.130.190:34281/ using document root <none> and password file <none>
I20260806 02:55:21.773473 24026 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:21.773576 24026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:21.773855 24026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:21.776175 24026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "d18ffcb4b39f4484a5d60913e5a58d1d"
format_stamp: "Formatted at 2026-08-06 02:55:21 on dist-test-slave-3kk6"
server_key: "8c2eb1f9e16e1d1e303a93740083b5a3"
server_key_iv: "25d16439c40fa5d7042935cbcce154ce"
server_key_version: "encryptionkey@0"
I20260806 02:55:21.776799 24026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "d18ffcb4b39f4484a5d60913e5a58d1d"
format_stamp: "Formatted at 2026-08-06 02:55:21 on dist-test-slave-3kk6"
server_key: "8c2eb1f9e16e1d1e303a93740083b5a3"
server_key_iv: "25d16439c40fa5d7042935cbcce154ce"
server_key_version: "encryptionkey@0"
I20260806 02:55:21.780725 24026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:55:21.783334 24045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:21.784641 24026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:21.784750 24026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "d18ffcb4b39f4484a5d60913e5a58d1d"
format_stamp: "Formatted at 2026-08-06 02:55:21 on dist-test-slave-3kk6"
server_key: "8c2eb1f9e16e1d1e303a93740083b5a3"
server_key_iv: "25d16439c40fa5d7042935cbcce154ce"
server_key_version: "encryptionkey@0"
I20260806 02:55:21.784847 24026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:21.789955 24026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:21.793346 24026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:21.793527 24026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:21.800595 24026 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:32973
I20260806 02:55:21.800710 24118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:32973 every 8 connection(s)
I20260806 02:55:21.801640 24026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:21.804325 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24026
I20260806 02:55:21.804486 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:21.804739 24119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:21.804790 25098 external_mini_cluster.cc:1428] Setting key a6049bd3cb4437341a10b95e2aa99f89
I20260806 02:55:21.810279 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:21.806241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50691 (local address 127.24.130.190:32973)
0806 02:55:21.806765 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:21.806782 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:21.806819 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:55:21.807552 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:21.807570 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:21.807855 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:21.808201 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:21.809057 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:21.809062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:21.809074 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:21.809085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:21.809140 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:21.809159 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:21.809310 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:21.809404 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 02:55:21.809573 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":152,"threads_started":1}
I20260806 02:55:21.811765 24119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d: Bootstrap starting.
I20260806 02:55:21.813275 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:21.814324 24119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:21.815320 24119 log.cc:824] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:21.817318 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:21.815570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45179 (local address 127.24.130.190:32973)
0806 02:55:21.815699 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:21.815705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:21.815885 (+ 180us) server_negotiation.cc:407] Connection header received
0806 02:55:21.816033 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:21.816038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:21.816074 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:21.816161 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:21.816919 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:21.816926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:21.816928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:55:21.816940 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:21.816966 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:55:21.816976 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:21.817070 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:21.817148 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:55:21.817190 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:55:21.817612 24119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d: No bootstrap required, opened a new log
I20260806 02:55:21.821357 24119 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 32973 } }
I20260806 02:55:21.821612 24119 raft_consensus.cc:374] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:21.821678 24119 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18ffcb4b39f4484a5d60913e5a58d1d, State: Initialized, Role: FOLLOWER
I20260806 02:55:21.822213 24119 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 32973 } }
I20260806 02:55:21.822419 24119 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:21.822497 24119 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:21.822603 24119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:21.823659 24119 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 32973 } }
I20260806 02:55:21.824030 24119 leader_election.cc:302] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18ffcb4b39f4484a5d60913e5a58d1d; no voters:
I20260806 02:55:21.824430 24119 leader_election.cc:288] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:21.824573 24134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:21.824841 24134 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [term 1 LEADER]: Becoming Leader. State: Replica: d18ffcb4b39f4484a5d60913e5a58d1d, State: Running, Role: LEADER
I20260806 02:55:21.825225 24134 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 32973 } }
I20260806 02:55:21.825726 24119 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:21.826987 24136 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 32973 } } }
I20260806 02:55:21.827126 24136 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:21.827541 24137 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18ffcb4b39f4484a5d60913e5a58d1d. Latest consensus state: current_term: 1 leader_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18ffcb4b39f4484a5d60913e5a58d1d" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 32973 } } }
I20260806 02:55:21.827570 24148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:21.827704 24137 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:21.831646 24148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:21.837957 24148 catalog_manager.cc:1347] Generated new cluster ID: 22861fcd518d48c0ad8e192826b2acb3
I20260806 02:55:21.838052 24148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:21.858564 24148 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:21.858968 24148 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:21.863174 24148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d18ffcb4b39f4484a5d60913e5a58d1d: Generated new TSK 0
I20260806 02:55:21.864063 24148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:21.872288 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24026
2026-08-06T02:55:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1635 ms)
[----------] 18 tests from AuthTokenIssuingTest (29623 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:21 dist-test-slave-3kk6 krb5kdc[24164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:21 dist-test-slave-3kk6 krb5kdc[24164](info): set up 2 sockets
Aug 06 02:55:21 dist-test-slave-3kk6 krb5kdc[24164](info): commencing operation
krb5kdc: starting...
W20260806 02:55:23.111510 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:23.250201 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44475
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44475,127.24.130.189:38055,127.24.130.188:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:23.368978 24183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:55:23.369415 24183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:23.369678 24183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:23.369801 24183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:23.374065 24183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:23.374183 24183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:23.374225 24183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:23.374267 24183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:23.374305 24183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:23.380095 24183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44475,127.24.130.189:38055,127.24.130.188:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:23.381338 24183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:23.382452 24183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:23.391664 24195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:23.391873 24183 server_base.cc:1034] running on GCE node
W20260806 02:55:23.391737 24193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.391664 24191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:23.392738 24183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:23.393936 24183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:23.395135 24183 hybrid_clock.cc:648] HybridClock initialized: now 1785984923395106 us; error 48 us; skew 500 ppm
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:23.398267 24183 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:23.399363 24183 webserver.cc:460] Webserver started at http://127.24.130.190:38687/ using document root <none> and password file <none>
I20260806 02:55:23.399969 24183 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:23.400035 24183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:23.400296 24183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:23.402503 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "1c7849d276fb44bab91e804eeb7c9052"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "15646db760564870b6026d3f89b62ab2"
server_key_iv: "a057cb76f19f8e45dea151b8cd796dcc"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.403196 24183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "1c7849d276fb44bab91e804eeb7c9052"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "15646db760564870b6026d3f89b62ab2"
server_key_iv: "a057cb76f19f8e45dea151b8cd796dcc"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.407171 24183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:23.409736 24202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:23.411123 24183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:55:23.411278 24183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "1c7849d276fb44bab91e804eeb7c9052"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "15646db760564870b6026d3f89b62ab2"
server_key_iv: "a057cb76f19f8e45dea151b8cd796dcc"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.411401 24183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:23.426492 24183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:23.429808 24183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:23.430052 24183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:23.437767 24183 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:44475
I20260806 02:55:23.438082 24274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:44475 every 8 connection(s)
I20260806 02:55:23.438913 24183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:23.441817 24276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:23.446084 24276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:23.448899 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24183
I20260806 02:55:23.449081 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:23.449424 25098 external_mini_cluster.cc:1428] Setting key 3f4e479d4a7c625a9c284715a39c0098
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.130.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.189@KRBTEST.COM" created.
I20260806 02:55:23.463156 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.448519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.24.130.190:44475)
0806 02:55:23.449265 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.449280 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.449332 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:55:23.450429 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.450438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.450736 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.450956 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.451865 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.452746 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.454313 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.454795 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.454968 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.457879 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.457884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.457896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.457916 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.459728 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.460275 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.460281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.460283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.460393 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.460724 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.460729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.460731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.461196 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.461338 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.461786 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.462044 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":182,"threads_started":1}
I20260806 02:55:23.463156 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.448518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44475 (local address 127.0.0.1:47618)
0806 02:55:23.449079 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.449138 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.450071 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.450965 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.450970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.451311 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.451648 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.451684 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.452890 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.452894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.454165 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.454181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.455000 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.455008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.455152 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.455816 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.455843 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.457670 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.459839 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.459850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.459861 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.460162 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.460494 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.460497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.460498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.460619 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.461321 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.461327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.461475 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.461719 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.462125 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":105,"thread_start_us":180,"threads_started":1}
I20260806 02:55:23.465977 24276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:23.467054 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.466567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39044)
0806 02:55:23.466880 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.466976 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
W20260806 02:55:23.467272 24205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
I20260806 02:55:23.468231 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.467999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39060)
0806 02:55:23.468117 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.468165 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:55:23.468379 24276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:38055: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab.
Entry for principal kudu/127.24.130.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab.
I20260806 02:55:23.479478 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:38055
--webserver_interface=127.24.130.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab
--principal=kudu/127.24.130.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44475,127.24.130.189:38055,127.24.130.188:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:23.517623 24276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } attempt: 1
I20260806 02:55:23.518541 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.518227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39062)
0806 02:55:23.518389 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.518461 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:55:23.519300 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.519081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39064)
0806 02:55:23.519189 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.519242 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 02:55:23.519454 24276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:38055: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
I20260806 02:55:23.587664 24276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } attempt: 2
I20260806 02:55:23.588548 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.588233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39070)
0806 02:55:23.588401 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.588470 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 02:55:23.589252 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.589042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39072)
0806 02:55:23.589146 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.589197 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:55:23.589387 24276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:38055: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38055: connect: Connection refused (error 111)
W20260806 02:55:23.597731 24288 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:55:23.598101 24288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:23.598181 24288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:23.598268 24288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:23.602438 24288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:23.602550 24288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:23.602592 24288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:23.602653 24288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:23.602722 24288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:23.608654 24288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44475,127.24.130.189:38055,127.24.130.188:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab
--principal=kudu/127.24.130.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.130.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:23.609869 24288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:23.611047 24288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:23.619879 24302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.619946 24299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.619890 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:23.620352 24288 server_base.cc:1034] running on GCE node
I20260806 02:55:23.620960 24288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:23.622138 24288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:23.623335 24288 hybrid_clock.cc:648] HybridClock initialized: now 1785984923623317 us; error 32 us; skew 500 ppm
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:23.628964 24288 init.cc:377] Logged in from keytab as kudu/127.24.130.189@KRBTEST.COM (short username kudu)
I20260806 02:55:23.630033 24288 webserver.cc:460] Webserver started at http://127.24.130.189:38119/ using document root <none> and password file <none>
I20260806 02:55:23.630695 24288 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:23.630813 24288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:23.631116 24288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:23.633342 24288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data/instance:
uuid: "a235404e3bb44e749307f4f18c117092"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "17621541fe638e497c4ccb5dff3cef96"
server_key_iv: "59ab11b55183fd88033c9522cf98afad"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.633978 24288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal/instance:
uuid: "a235404e3bb44e749307f4f18c117092"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "17621541fe638e497c4ccb5dff3cef96"
server_key_iv: "59ab11b55183fd88033c9522cf98afad"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.638079 24288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:55:23.640868 24310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:23.642150 24288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:55:23.642310 24288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal
uuid: "a235404e3bb44e749307f4f18c117092"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "17621541fe638e497c4ccb5dff3cef96"
server_key_iv: "59ab11b55183fd88033c9522cf98afad"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.642450 24288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:23.663055 24288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:23.666281 24288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:23.666527 24288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:23.674276 24288 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.189:38055
I20260806 02:55:23.674621 24384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.189:38055 every 8 connection(s)
I20260806 02:55:23.675417 24288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
I20260806 02:55:23.677906 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24288
I20260806 02:55:23.678046 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-1/wal/instance
I20260806 02:55:23.678398 25098 external_mini_cluster.cc:1428] Setting key 3d483f6bd449a4635666e177d516c5bc
I20260806 02:55:23.678574 24385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:23.680563 24276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } attempt: 3
I20260806 02:55:23.683736 24385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.130.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.188@KRBTEST.COM" created.
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:23.697243 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.681198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39086)
0806 02:55:23.681448 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.681468 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.681590 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.684365 (+ 2775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.684370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.684615 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.684799 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.684812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.686499 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.686502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.687369 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.687382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.688224 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.688228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.688348 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.688968 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.688990 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.692462 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.694919 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.694923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.694926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.695261 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.695716 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.695720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.695721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.695827 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.696662 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.696664 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.696751 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.696864 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.697041 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1}
I20260806 02:55:23.697243 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.685879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.24.130.190:44475)
0806 02:55:23.685993 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.685998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.686421 (+ 423us) server_negotiation.cc:407] Connection header received
0806 02:55:23.686618 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.686621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.686694 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.686745 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.687708 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.688534 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.689525 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.690374 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.690511 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.693357 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.693361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.693363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.693397 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.695256 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.695896 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.695900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.695901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.695983 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.696250 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.696255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.696256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.696453 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.696568 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.697028 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.697132 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:55:23.698508 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.682033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.24.130.189:38055)
0806 02:55:23.682548 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.682573 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.682619 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:55:23.683548 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.683572 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.684004 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.684399 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.684984 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.686334 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.687534 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.688084 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.688259 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.692657 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.692668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.692687 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.692721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.694801 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.695392 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.695400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.695405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.695596 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.695943 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.695947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.695948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.696550 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.696694 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.697000 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.697339 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":346,"spinlock_wait_cycles":8576,"thread_start_us":153,"threads_started":1}
I20260806 02:55:23.698506 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.685852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44475 (local address 127.0.0.1:47626)
0806 02:55:23.686285 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.686338 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.686516 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.686931 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.686937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.687380 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.687560 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.687567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.688703 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.688709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.689388 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.689399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.690569 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.690574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.690731 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.691225 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.691242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.693182 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.695386 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.695396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.695407 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.695754 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.696078 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.696080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.696081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.696152 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.696568 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.696576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.696750 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.697268 (+ 518us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.697485 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":2944,"thread_start_us":141,"threads_started":1}
I20260806 02:55:23.700565 24385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:23.700657 24276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:23.701388 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.701116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55472)
0806 02:55:23.701246 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.701298 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:55:23.702509 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.701961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55482)
0806 02:55:23.702337 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.702392 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
W20260806 02:55:23.702643 24276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:36285: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:55:23.709801 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.701453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39098)
0806 02:55:23.701624 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.701639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.701737 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.702056 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.702061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.702261 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.702374 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.702383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.703276 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.703282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.703834 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.703843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.704493 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.704496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.704559 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.705001 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.705012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.706349 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.708339 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.708342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.708343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.708532 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.708869 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.708871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.708872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.708956 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.709348 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.709350 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.709432 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.709516 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.709680 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 02:55:23.709945 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.701509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.24.130.189:38055)
0806 02:55:23.701616 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.701620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.701687 (+ 67us) server_negotiation.cc:407] Connection header received
0806 02:55:23.701840 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.701868 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.701923 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.702009 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.702512 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.703151 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.703949 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.704388 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.704473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.706478 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.706481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.706483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.706505 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.708239 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.708666 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.708672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.708676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.708772 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.709023 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.709025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.709026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.709261 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.709338 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.709551 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.709784 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":43}
I20260806 02:55:23.710599 24385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab.
Entry for principal kudu/127.24.130.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab.
I20260806 02:55:23.711462 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.711148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55494)
0806 02:55:23.711276 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.711382 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:55:23.711846 24314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
I20260806 02:55:23.711911 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:36285
--webserver_interface=127.24.130.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab
--principal=kudu/127.24.130.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:44475,127.24.130.189:38055,127.24.130.188:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:23.712759 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.712548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55506)
0806 02:55:23.712658 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.712711 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:55:23.712925 24385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:36285: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
I20260806 02:55:23.733861 24276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } attempt: 1
I20260806 02:55:23.734892 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.734500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55510)
0806 02:55:23.734741 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.734807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:55:23.735857 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.735558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55516)
0806 02:55:23.735731 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.735794 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 02:55:23.736090 24276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:36285: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
I20260806 02:55:23.762212 24385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } attempt: 1
I20260806 02:55:23.763269 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.762896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55520)
0806 02:55:23.763090 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.763173 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:55:23.764097 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.763858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55532)
0806 02:55:23.763974 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.764031 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:55:23.764299 24385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:36285: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
I20260806 02:55:23.811296 24276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } attempt: 2
I20260806 02:55:23.812223 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.811900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55534)
0806 02:55:23.812058 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.812140 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:55:23.813017 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.812790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55546)
0806 02:55:23.812902 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.812960 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:55:23.813158 24276 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:36285: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
W20260806 02:55:23.830951 24397 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:55:23.831284 24397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:23.831357 24397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:23.831445 24397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:55:23.832490 24385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } attempt: 2
I20260806 02:55:23.833390 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.833038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55548)
0806 02:55:23.833222 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.833298 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:55:23.834195 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.833963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55554)
0806 02:55:23.834078 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.834130 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:55:23.834376 24385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:36285: Network error: Client connection negotiation failed: client connection to 127.24.130.188:36285: connect: Connection refused (error 111)
W20260806 02:55:23.836104 24397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:23.836159 24397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:23.836179 24397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:23.836233 24397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:23.836268 24397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:23.841671 24397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:44475,127.24.130.189:38055,127.24.130.188:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab
--principal=kudu/127.24.130.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.130.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:23.842860 24397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:23.843907 24397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:55:23.853080 24397 server_base.cc:1034] running on GCE node
W20260806 02:55:23.853030 24403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.853034 24404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.853030 24406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:23.853966 24397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:23.855290 24397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:23.856494 24397 hybrid_clock.cc:648] HybridClock initialized: now 1785984923856465 us; error 44 us; skew 500 ppm
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:23.859627 24397 init.cc:377] Logged in from keytab as kudu/127.24.130.188@KRBTEST.COM (short username kudu)
I20260806 02:55:23.860534 24397 webserver.cc:460] Webserver started at http://127.24.130.188:40195/ using document root <none> and password file <none>
I20260806 02:55:23.861065 24397 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:23.861126 24397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:23.861299 24397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:23.863410 24397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data/instance:
uuid: "36506e19ddac4d6ba69162ac797f4150"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "53276e945674bdef7016043ca02919ba"
server_key_iv: "57a253a518abded532b8a5c6f3d27a34"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.863953 24397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal/instance:
uuid: "36506e19ddac4d6ba69162ac797f4150"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "53276e945674bdef7016043ca02919ba"
server_key_iv: "57a253a518abded532b8a5c6f3d27a34"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.867774 24397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:23.870211 24418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:23.871517 24397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:55:23.871660 24397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal
uuid: "36506e19ddac4d6ba69162ac797f4150"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-3kk6"
server_key: "53276e945674bdef7016043ca02919ba"
server_key_iv: "57a253a518abded532b8a5c6f3d27a34"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.871793 24397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:23.886945 24397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:23.890352 24397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:23.890576 24397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:23.898288 24397 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.188:36285
I20260806 02:55:23.898626 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.188:36285 every 8 connection(s)
I20260806 02:55:23.899473 24397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
I20260806 02:55:23.900180 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24397
I20260806 02:55:23.900336 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984624291587-25098-0/minicluster-data/master-2/wal/instance
I20260806 02:55:23.900641 25098 external_mini_cluster.cc:1428] Setting key 790d44be7c5e97c55a3c2e168a033390
I20260806 02:55:23.902931 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:23.907480 24490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:55:23.912322 24276 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } attempt: 3
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:23.917819 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.902166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.24.130.188:36285)
0806 02:55:23.902604 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.902621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.902704 (+ 83us) server_negotiation.cc:407] Connection header received
0806 02:55:23.903470 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.903488 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.903821 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.904120 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.905146 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.906157 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.907363 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.907827 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.908066 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.911169 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.911174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.911182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.911210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.913673 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.914245 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.914251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.914258 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.914431 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.915093 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.915096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.915098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.915633 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.915822 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.916238 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.916644 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":158,"threads_started":1}
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:55:23.921934 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.909619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:44475 (local address 127.0.0.1:47636)
0806 02:55:23.910075 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.910134 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.910277 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.910632 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.910638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.911170 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.911299 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.911306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.912352 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.912357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.913541 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.913549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.914212 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.914216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.914330 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.915187 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.915200 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.917037 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.919499 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.919506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.919511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.919823 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.920211 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.920217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.920234 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.920349 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.920863 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.920870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.921029 (+ 159us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.921513 (+ 484us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.921736 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":162,"threads_started":1}
I20260806 02:55:23.922048 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.909686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.24.130.190:44475)
0806 02:55:23.909854 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.909859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.910231 (+ 372us) server_negotiation.cc:407] Connection header received
0806 02:55:23.910381 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.910383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.910435 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.910525 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.911440 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.912196 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.913669 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.914079 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.914213 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.917194 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.917198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.917200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.917238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.919381 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.919946 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.919954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.919956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.920070 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.920459 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.920464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.920467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.920740 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.920863 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.921280 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.921410 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:55:23.923262 24490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:23.923754 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.912786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55566)
0806 02:55:23.912915 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.912935 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.913031 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.913466 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.913470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.913686 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.913819 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.913831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.914766 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.914769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.915584 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.915595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.916237 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.916244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.916355 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.916874 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.916888 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.918408 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.921453 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.921457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.921459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.921800 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.922576 (+ 776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.922582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.922586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.922732 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.923222 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.923225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.923316 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.923421 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.923557 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 02:55:23.924294 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.912836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.24.130.188:36285)
0806 02:55:23.913184 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.913193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.913208 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:55:23.913264 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.913267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.913309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.913408 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.913943 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.914623 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.915729 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.916070 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.916593 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.918563 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.918569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.918572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.918600 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.921305 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.922330 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.922333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.922335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.922438 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.922865 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.922868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.922870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.923103 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.923214 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.923960 (+ 746us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.924098 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260806 02:55:23.925565 24385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } attempt: 3
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:23.929136 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052: Bootstrap starting.
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:55:23.932773 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.921151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.24.130.190:44475)
0806 02:55:23.921419 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.921432 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.921446 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:55:23.921545 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.921547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.921591 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.921685 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.922733 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.923314 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.924543 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.924814 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.924952 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.928157 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.928164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.928167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.928200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.930344 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.931098 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.931103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.931106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.931213 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.931649 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.931654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.931658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.931952 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.932193 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.932390 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.932541 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260806 02:55:23.933135 24276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052: Neither blocks nor log segments found. Creating new log.
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:55:23.934194 24276 log.cc:824] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:23.937115 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.923981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.24.130.189:38055)
0806 02:55:23.924102 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.924107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.924183 (+ 76us) server_negotiation.cc:407] Connection header received
0806 02:55:23.924304 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.924307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.924357 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.924450 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.925519 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.926152 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.927289 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.927572 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.927687 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.930237 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.930242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.930244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.930279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.932475 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.934601 (+ 2126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.934607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.934612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.934745 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.935137 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.935141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.935143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.935411 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.935520 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.936789 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.936914 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:55:23.937886 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.923843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38055 (local address 127.0.0.1:39102)
0806 02:55:23.924124 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.924138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.924219 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.925081 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.925085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.925266 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.925370 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.925376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.926299 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.926303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.927138 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.927148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.927686 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.927690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.927793 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.928295 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.928313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.930043 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.934131 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.934135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.934137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.934451 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.934902 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.934908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.934911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.935017 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.936378 (+ 1361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.936382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.936537 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.937522 (+ 985us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.937686 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260806 02:55:23.938174 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052: No bootstrap required, opened a new log
I20260806 02:55:23.938951 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.926495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55572)
0806 02:55:23.926713 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.926738 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.926818 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.928439 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.928442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.928637 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.928753 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.928764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.929849 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.929853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.930641 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.930679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.931346 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.931354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.931457 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.932743 (+ 1286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.932765 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.934966 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.937273 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.937278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.937281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.937534 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.937848 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.937851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.937853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.937961 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.938426 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.938429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.938521 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.938605 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.938793 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 02:55:23.938942 24490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:23.939535 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.926537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.24.130.188:36285)
0806 02:55:23.928183 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.928188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.928203 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:55:23.928245 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.928248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.928290 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.928877 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.928968 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.929710 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.930825 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.931208 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.931330 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.935152 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.935156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.935159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.935185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.937142 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.937658 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.937661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.937664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.937742 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.938088 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.938091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.938092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.938306 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.939188 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.939266 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.939395 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562}
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:55:23.942868 24385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: Bootstrap starting.
I20260806 02:55:23.944168 24276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.944449 24276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7849d276fb44bab91e804eeb7c9052, State: Initialized, Role: FOLLOWER
I20260806 02:55:23.944857 24276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.946170 24385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:23.946552 24276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:23.947326 24385 log.cc:824] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:23.948028 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.934320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.24.130.189:38055)
0806 02:55:23.935295 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.935302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.935313 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:55:23.935343 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.935345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.935382 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.935513 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.936358 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.937367 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.938533 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.938948 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.939060 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.942532 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.942535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.942538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.942563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.944711 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.945417 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.945420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.945422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.945516 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.945871 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.945874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.945876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.946142 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.947577 (+ 1435us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.947665 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.947820 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":178,"threads_started":1}
Aug 06 02:55:23 dist-test-slave-3kk6 krb5kdc[24164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:55:23.949934 24385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: No bootstrap required, opened a new log
W20260806 02:55:23.953343 24514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:23.953452 24514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:23.953799 24500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:23.953974 24500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:23.954063 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.940177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.24.130.188:36285)
0806 02:55:23.942877 (+ 2700us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.942884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.942905 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:55:23.942956 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.942960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.943021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.943117 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.943763 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.944560 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.945874 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.946208 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.946311 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.950110 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:23.950114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:23.950117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:23.950141 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:23.952053 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.952603 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.952607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.952609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.952695 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:23.953003 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:23.953006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:23.953008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:23.953247 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:23.953326 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:23.953737 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 02:55:23.953879 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2588}
I20260806 02:55:23.954273 24385 raft_consensus.cc:348] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.954535 24385 raft_consensus.cc:729] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a235404e3bb44e749307f4f18c117092, State: Initialized, Role: FOLLOWER
I20260806 02:55:23.954635 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.939454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:36285 (local address 127.0.0.1:55586)
0806 02:55:23.939754 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:23.939771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:23.939860 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:23.943247 (+ 3387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:23.943250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:23.943472 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:23.943626 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.943635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.944784 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.944788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.945736 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:23.945749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.946470 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.946474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:23.946589 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:23.947235 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:23.947252 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:23.949952 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:23.952189 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.952192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.952194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.952485 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.952813 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:23.952816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:23.952818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:23.952907 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:23.953401 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:23.953403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:23.953508 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:55:23.954294 (+ 786us) client_negotiation.cc:240] Negotiation successful
0806 02:55:23.954437 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
I20260806 02:55:23.957540 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:23.958031 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: Bootstrap starting.
I20260806 02:55:23.960930 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:23.961969 24490 log.cc:824] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:23.962810 24385 consensus_queue.cc:260] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.959029 24240 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1c7849d276fb44bab91e804eeb7c9052"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47644
I20260806 02:55:23.963503 24240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:55:23.963637 24240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:23.964345 24385 sys_catalog.cc:564] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:23.964671 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: No bootstrap required, opened a new log
I20260806 02:55:23.965124 24240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.967279 24345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c7849d276fb44bab91e804eeb7c9052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a235404e3bb44e749307f4f18c117092"
I20260806 02:55:23.968930 24345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:23.970069 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.970310 24345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7849d276fb44bab91e804eeb7c9052 in term 1.
I20260806 02:55:23.970371 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36506e19ddac4d6ba69162ac797f4150, State: Initialized, Role: FOLLOWER
I20260806 02:55:23.970757 24205 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7849d276fb44bab91e804eeb7c9052, a235404e3bb44e749307f4f18c117092; no voters:
I20260806 02:55:23.970784 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.971060 24500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:23.971262 24500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7849d276fb44bab91e804eeb7c9052, State: Running, Role: LEADER
I20260806 02:55:23.971562 24500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } }
I20260806 02:55:23.972359 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:23.972415 24459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c7849d276fb44bab91e804eeb7c9052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36506e19ddac4d6ba69162ac797f4150"
I20260806 02:55:23.972620 24459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:23.973286 24500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7849d276fb44bab91e804eeb7c9052. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:23.973399 24500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:23.974000 24544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:23.974195 24459 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7849d276fb44bab91e804eeb7c9052 in term 1.
I20260806 02:55:23.978818 24183 master_runner.cc:263] No leader master found from master 1c7849d276fb44bab91e804eeb7c9052
I20260806 02:55:23.978818 24288 master_runner.cc:263] No leader master found from master a235404e3bb44e749307f4f18c117092
I20260806 02:55:23.978894 24288 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:55:23.978893 24183 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:55:23.979020 24240 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [CANDIDATE]: Term 1 election: Requested vote from peers a235404e3bb44e749307f4f18c117092 (127.24.130.189:38055), 36506e19ddac4d6ba69162ac797f4150 (127.24.130.188:36285)
W20260806 02:55:23.980916 24531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:23.981000 24531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:23.981343 24544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:23.983520 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:23.983649 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:23.987169 24456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [term 1 FOLLOWER]: Refusing update from remote peer 1c7849d276fb44bab91e804eeb7c9052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:23.988041 24533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:23.992949 24345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [term 1 FOLLOWER]: Refusing update from remote peer 1c7849d276fb44bab91e804eeb7c9052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:23.993546 24533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:55:23.995961 24555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:23.996117 24555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:23.996410 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:23.996625 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:23.996785 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7849d276fb44bab91e804eeb7c9052. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:23.996960 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:23.997207 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7849d276fb44bab91e804eeb7c9052. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:23.997339 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:24.004748 24533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:24.004935 24533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:24.005066 24564 mvcc.cc:205] Tried to move back new op lower bound from 7315394248643596288 to 7315394248590888960. Current Snapshot: MvccSnapshot[applied={T|T < 7315394248643596288}]
I20260806 02:55:24.005220 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:24.005368 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:24.005738 24570 mvcc.cc:205] Tried to move back new op lower bound from 7315394248643596288 to 7315394248590888960. Current Snapshot: MvccSnapshot[applied={T|T < 7315394248643596288}]
I20260806 02:55:24.008577 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.982609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.24.130.188:36285)
0806 02:55:23.982752 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.982757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.997796 (+ 15039us) server_negotiation.cc:407] Connection header received
0806 02:55:23.997928 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.997932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.997993 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.998086 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.999289 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.000167 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.001471 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.001853 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.001943 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.004043 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.004047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.004050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.004080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.006399 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.007068 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.007072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.007074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.007170 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.007514 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.007518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.007520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.007748 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.007870 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.008264 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.008404 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:55:24.009462 24544 catalog_manager.cc:1347] Generated new cluster ID: 5f45b43957564fc3b839c090cd624b20
I20260806 02:55:24.009526 24544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:24.012677 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.982007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.24.130.190:44475)
0806 02:55:23.982442 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.982450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.986896 (+ 4446us) server_negotiation.cc:407] Connection header received
0806 02:55:23.987120 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.987124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.987179 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.987250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:23.988674 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.989614 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.991053 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:23.991580 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:23.991706 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.008449 (+ 16743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.008456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.008459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.008491 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.010339 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.011410 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.011414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.011416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.011506 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.011803 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.011806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.011808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.012004 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.012104 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.012412 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.012534 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260806 02:55:24.015808 24500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:24.015966 24500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:24.017167 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c7849d276fb44bab91e804eeb7c9052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7849d276fb44bab91e804eeb7c9052" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 44475 } } peers { permanent_uuid: "a235404e3bb44e749307f4f18c117092" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38055 } } peers { permanent_uuid: "36506e19ddac4d6ba69162ac797f4150" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 36285 } } }
I20260806 02:55:24.017328 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:24.018684 24573 mvcc.cc:205] Tried to move back new op lower bound from 7315394248643596288 to 7315394248590888960. Current Snapshot: MvccSnapshot[applied={T|T < 7315394248643596288}]
I20260806 02:55:24.019621 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:23.986197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.24.130.189:38055)
0806 02:55:23.986331 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:23.986335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:23.998118 (+ 11783us) server_negotiation.cc:407] Connection header received
0806 02:55:23.998241 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:23.998245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:23.998296 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:23.998377 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.006774 (+ 8397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.007569 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.012926 (+ 5357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.013315 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.013400 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.015949 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.015955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.015957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.015987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.017854 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.018304 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.018307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.018309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.018385 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.018693 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.018696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.018698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.018893 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.019013 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.019334 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.019475 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:55:24.058771 24544 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:24.061079 24544 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:24.073607 24544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7849d276fb44bab91e804eeb7c9052: Generated new TSK 0
I20260806 02:55:24.074525 24544 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:55:24.129767 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.127332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.24.130.189:38055)
0806 02:55:24.127537 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.127543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.128625 (+ 1082us) server_negotiation.cc:407] Connection header received
0806 02:55:24.128795 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.128799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.128857 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.128961 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.129583 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39124: BlockingRecv error: recv got EOF from 127.0.0.1:39124 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 02:55:24.130786 24496 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.127789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.24.130.190:44475)
0806 02:55:24.127919 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.127926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.129959 (+ 2033us) server_negotiation.cc:407] Connection header received
0806 02:55:24.130014 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.130017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.130072 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.130194 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.130625 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47664: BlockingRecv error: recv got EOF from 127.0.0.1:47664 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 02:55:24.132340 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.128058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.24.130.188:36285)
0806 02:55:24.128199 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.128207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.131368 (+ 3161us) server_negotiation.cc:407] Connection header received
0806 02:55:24.131555 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.131559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.131610 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.131663 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.132219 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55600: BlockingRecv error: recv got EOF from 127.0.0.1:55600 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 02:55:24.144697 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.139268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.24.130.189:38055)
0806 02:55:24.139392 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.139396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.143458 (+ 4062us) server_negotiation.cc:407] Connection header received
0806 02:55:24.143740 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.143744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.143825 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.143883 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.144566 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39130: BlockingRecv error: recv got EOF from 127.0.0.1:39130 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:55:24.153393 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.151610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.24.130.189:38055)
0806 02:55:24.151789 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.151794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.152227 (+ 433us) server_negotiation.cc:407] Connection header received
0806 02:55:24.152465 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.152469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.152524 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.152581 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.153234 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39146: BlockingRecv error: recv got EOF from 127.0.0.1:39146 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 02:55:24.159994 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.158197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.24.130.189:38055)
0806 02:55:24.158316 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.158321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.158911 (+ 590us) server_negotiation.cc:407] Connection header received
0806 02:55:24.159139 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.159142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.159199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.159259 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.159864 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39152: BlockingRecv error: recv got EOF from 127.0.0.1:39152 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:55:24.170426 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.167485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.24.130.189:38055)
0806 02:55:24.167636 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.167640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.169252 (+ 1612us) server_negotiation.cc:407] Connection header received
0806 02:55:24.169526 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.169530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.169591 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.169653 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.170300 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39168: BlockingRecv error: recv got EOF from 127.0.0.1:39168 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 02:55:24.183132 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.181332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.24.130.189:38055)
0806 02:55:24.181455 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.181459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.182080 (+ 621us) server_negotiation.cc:407] Connection header received
0806 02:55:24.182280 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.182284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.182336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.182388 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.183014 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39176: BlockingRecv error: recv got EOF from 127.0.0.1:39176 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:55:24.204030 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.202113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.24.130.189:38055)
0806 02:55:24.202253 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.202258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.203038 (+ 780us) server_negotiation.cc:407] Connection header received
0806 02:55:24.203234 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.203237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.203288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.203387 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.203894 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39188: BlockingRecv error: recv got EOF from 127.0.0.1:39188 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:55:24.239893 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.238489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.24.130.189:38055)
0806 02:55:24.238669 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.238674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.238854 (+ 180us) server_negotiation.cc:407] Connection header received
0806 02:55:24.238970 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.238974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.239040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.239159 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.239731 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39198: BlockingRecv error: recv got EOF from 127.0.0.1:39198 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:55:24.307502 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.306128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.24.130.189:38055)
0806 02:55:24.306319 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.306324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.306495 (+ 171us) server_negotiation.cc:407] Connection header received
0806 02:55:24.306727 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.306730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.306777 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.306877 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.307386 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39212: BlockingRecv error: recv got EOF from 127.0.0.1:39212 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260806 02:55:24.439644 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.438162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.24.130.189:38055)
0806 02:55:24.438326 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.438331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.438535 (+ 204us) server_negotiation.cc:407] Connection header received
0806 02:55:24.438804 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.438808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.438860 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.438929 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.439519 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39228: BlockingRecv error: recv got EOF from 127.0.0.1:39228 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:55:24.699327 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:24.698177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.24.130.189:38055)
0806 02:55:24.698351 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.698355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.698425 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:55:24.698568 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.698570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.698619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.698752 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.699220 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39238: BlockingRecv error: recv got EOF from 127.0.0.1:39238 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:55:24.982364 24531 catalog_manager.cc:1259] Loaded cluster ID: 5f45b43957564fc3b839c090cd624b20
I20260806 02:55:24.982471 24531 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: loading cluster ID for follower catalog manager: success
I20260806 02:55:24.984335 24531 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: acquiring CA information for follower catalog manager: success
I20260806 02:55:24.985673 24531 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a235404e3bb44e749307f4f18c117092: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:55:24.997706 24555 catalog_manager.cc:1259] Loaded cluster ID: 5f45b43957564fc3b839c090cd624b20
I20260806 02:55:24.997826 24555 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: loading cluster ID for follower catalog manager: success
I20260806 02:55:24.999795 24555 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: acquiring CA information for follower catalog manager: success
I20260806 02:55:25.001070 24555 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36506e19ddac4d6ba69162ac797f4150: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:55:25.217835 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.213721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.24.130.189:38055)
0806 02:55:25.213979 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.213983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.214001 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:55:25.214122 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.214124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.214324 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.214454 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:25.214839 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.215461 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.216398 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.216625 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.216701 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.217001 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:25.217454 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:25.217548 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.217621 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.217674 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260806 02:55:25.225327 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.221483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.24.130.188:36285)
0806 02:55:25.221738 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.221742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.221828 (+ 86us) server_negotiation.cc:407] Connection header received
0806 02:55:25.221992 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.221994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.222120 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.222227 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:25.222488 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.223039 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.223892 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.224109 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.224186 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.224470 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:25.224906 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:25.224999 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.225056 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.225105 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260806 02:55:25.232733 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.228567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.24.130.189:38055)
0806 02:55:25.228841 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.228845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.229245 (+ 400us) server_negotiation.cc:407] Connection header received
0806 02:55:25.229457 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.229465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.229535 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.229610 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:25.229960 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.230506 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.231462 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.231799 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.231908 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.232229 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:25.232324 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:25.232399 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.232514 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.232585 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260806 02:55:25.232766 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.228693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.24.130.188:36285)
0806 02:55:25.228811 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.228816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.229214 (+ 398us) server_negotiation.cc:407] Connection header received
0806 02:55:25.229404 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.229408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.229458 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.229548 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:25.229809 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.230367 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.231664 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.231937 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.232039 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.232305 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:25.232380 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:25.232443 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.232600 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.232649 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:55:25.234383 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.228528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.24.130.190:44475)
0806 02:55:25.229006 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.229011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.229103 (+ 92us) server_negotiation.cc:407] Connection header received
0806 02:55:25.229296 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.229299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.229491 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.229612 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:25.230674 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.231491 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.232633 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.233152 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.233254 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.233673 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:25.234027 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:25.234105 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.234214 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.234267 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":75,"threads_started":1}
I20260806 02:55:25.237933 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24183
I20260806 02:55:25.253749 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24288
I20260806 02:55:25.267520 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24397
2026-08-06T02:55:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3390 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:25 dist-test-slave-3kk6 krb5kdc[24685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:25 dist-test-slave-3kk6 krb5kdc[24685](info): set up 2 sockets
Aug 06 02:55:25 dist-test-slave-3kk6 krb5kdc[24685](info): commencing operation
krb5kdc: starting...
W20260806 02:55:26.536095 25098 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/test-user.keytab.
Aug 06 02:55:26 dist-test-slave-3kk6 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.130.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
Entry for principal kudu/127.24.130.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab.
I20260806 02:55:26.678020 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36421
--webserver_interface=127.24.130.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36421,127.24.130.189:38489,127.24.130.188:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:26.800340 24702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:55:26.800725 24702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:26.800751 24702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:26.800792 24702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:26.804982 24702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:26.805119 24702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:26.805160 24702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:26.805199 24702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:26.805253 24702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:26.810981 24702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36421,127.24.130.189:38489,127.24.130.188:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.190.keytab
--principal=kudu/127.24.130.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.190:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.130.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:26.812304 24702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:26.813469 24702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:26.822824 24708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:26.822824 24710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:26.822824 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:26.822988 24702 server_base.cc:1034] running on GCE node
I20260806 02:55:26.823894 24702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:26.825163 24702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:26.826376 24702 hybrid_clock.cc:648] HybridClock initialized: now 1785984926826366 us; error 39 us; skew 500 ppm
Aug 06 02:55:26 dist-test-slave-3kk6 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:26.829383 24702 init.cc:377] Logged in from keytab as kudu/127.24.130.190@KRBTEST.COM (short username kudu)
I20260806 02:55:26.830507 24702 webserver.cc:460] Webserver started at http://127.24.130.190:35451/ using document root <none> and password file <none>
I20260806 02:55:26.831187 24702 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:26.831291 24702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:26.831549 24702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:26.833901 24702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data/instance:
uuid: "41bfd54e97cd4023a84f49ea485dfe3e"
format_stamp: "Formatted at 2026-08-06 02:55:26 on dist-test-slave-3kk6"
server_key: "983d477a1865630d43c820bb71431847"
server_key_iv: "6012837759ba200a5123bb7a2b4914b9"
server_key_version: "encryptionkey@0"
I20260806 02:55:26.834584 24702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal/instance:
uuid: "41bfd54e97cd4023a84f49ea485dfe3e"
format_stamp: "Formatted at 2026-08-06 02:55:26 on dist-test-slave-3kk6"
server_key: "983d477a1865630d43c820bb71431847"
server_key_iv: "6012837759ba200a5123bb7a2b4914b9"
server_key_version: "encryptionkey@0"
I20260806 02:55:26.840137 24702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 02:55:26.843044 24720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:26.844414 24702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:55:26.844590 24702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal
uuid: "41bfd54e97cd4023a84f49ea485dfe3e"
format_stamp: "Formatted at 2026-08-06 02:55:26 on dist-test-slave-3kk6"
server_key: "983d477a1865630d43c820bb71431847"
server_key_iv: "6012837759ba200a5123bb7a2b4914b9"
server_key_version: "encryptionkey@0"
I20260806 02:55:26.844736 24702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:26.861692 24702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:26.865231 24702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:26.865473 24702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:26.873441 24702 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.190:36421
I20260806 02:55:26.873751 24787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.190:36421 every 8 connection(s)
I20260806 02:55:26.874624 24702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:26.876579 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24702
I20260806 02:55:26.876730 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-0/wal/instance
I20260806 02:55:26.877067 25098 external_mini_cluster.cc:1428] Setting key b2176d50324f492769e20a915b69326d
I20260806 02:55:26.878028 24789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:26.882413 24789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.130.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.189@KRBTEST.COM" created.
Aug 06 02:55:26 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:26.900565 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:26.885277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36421 (local address 127.0.0.1:41768)
0806 02:55:26.885893 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:26.885943 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:26.887120 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:26.888162 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:26.888168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:26.888645 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:26.888928 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:26.888943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:26.890706 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:26.890708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:26.891480 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:26.891490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:26.892326 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:26.892331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:26.892447 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:26.893150 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:26.893168 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:26.894865 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:26.897125 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:26.897135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:26.897147 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:26.897414 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:26.897780 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:26.897783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:26.897785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:26.897914 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:26.898605 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:26.898611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:26.898807 (+ 196us) client_negotiation.cc:769] Sending connection context
0806 02:55:26.898992 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:55:26.899442 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":121,"thread_start_us":182,"threads_started":1}
I20260806 02:55:26.900566 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:26.886436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.24.130.190:36421)
0806 02:55:26.886958 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:26.886971 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:26.887027 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:55:26.887547 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:26.887551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:26.887861 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:26.888152 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:26.889148 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:26.890518 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:26.891625 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:26.892088 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:26.892298 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:26.895041 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:26.895049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:26.895071 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:26.895094 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:26.896973 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:26.897551 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:26.897556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:26.897558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:26.897673 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:26.898019 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:26.898024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:26.898030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:26.898472 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:26.898627 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:26.899086 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 02:55:26.899384 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":389,"thread_start_us":93,"threads_started":1}
I20260806 02:55:26.904122 24789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:26.905037 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:26.904727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38952)
0806 02:55:26.904868 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:26.904963 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:55:26.905293 24721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
I20260806 02:55:26.906335 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:26.906104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38968)
0806 02:55:26.906227 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:26.906279 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:55:26.906514 24789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:38489: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab.
Entry for principal kudu/127.24.130.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab.
I20260806 02:55:26.909648 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:38489
--webserver_interface=127.24.130.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab
--principal=kudu/127.24.130.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36421,127.24.130.189:38489,127.24.130.188:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:26.955835 24789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } attempt: 1
I20260806 02:55:26.956990 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:26.956610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38970)
0806 02:55:26.956816 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:26.956898 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 02:55:26.957881 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:26.957625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38976)
0806 02:55:26.957755 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:26.957811 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:55:26.958084 24789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:38489: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
I20260806 02:55:27.026304 24789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } attempt: 2
I20260806 02:55:27.027384 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.027015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38980)
0806 02:55:27.027212 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.027296 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 02:55:27.028191 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.027949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38982)
0806 02:55:27.028069 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.028128 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:55:27.028337 24789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.189:38489: Network error: Client connection negotiation failed: client connection to 127.24.130.189:38489: connect: Connection refused (error 111)
W20260806 02:55:27.032187 24797 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:55:27.032577 24797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:27.032644 24797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:27.032725 24797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:27.037130 24797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:27.037249 24797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:27.037293 24797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:27.037331 24797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:27.037364 24797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:27.043449 24797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36421,127.24.130.189:38489,127.24.130.188:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.189.keytab
--principal=kudu/127.24.130.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.189:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.130.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:27.044834 24797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:27.046020 24797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:27.055774 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:27.055774 24803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:27.055965 24797 server_base.cc:1034] running on GCE node
W20260806 02:55:27.056191 24804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:27.056882 24797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:27.058149 24797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:27.059351 24797 hybrid_clock.cc:648] HybridClock initialized: now 1785984927059321 us; error 51 us; skew 500 ppm
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:27.062582 24797 init.cc:377] Logged in from keytab as kudu/127.24.130.189@KRBTEST.COM (short username kudu)
I20260806 02:55:27.063786 24797 webserver.cc:460] Webserver started at http://127.24.130.189:36129/ using document root <none> and password file <none>
I20260806 02:55:27.064443 24797 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:27.064550 24797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:27.064815 24797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:27.067214 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data/instance:
uuid: "f6fa1323664f443f99f13242e07d1099"
format_stamp: "Formatted at 2026-08-06 02:55:27 on dist-test-slave-3kk6"
server_key: "51aab40be2a8f359decadf590973a638"
server_key_iv: "14a4c206630ab61fe5c285a1808e2f55"
server_key_version: "encryptionkey@0"
I20260806 02:55:27.067860 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal/instance:
uuid: "f6fa1323664f443f99f13242e07d1099"
format_stamp: "Formatted at 2026-08-06 02:55:27 on dist-test-slave-3kk6"
server_key: "51aab40be2a8f359decadf590973a638"
server_key_iv: "14a4c206630ab61fe5c285a1808e2f55"
server_key_version: "encryptionkey@0"
I20260806 02:55:27.072695 24797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:55:27.075695 24817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:27.077172 24797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:55:27.077373 24797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal
uuid: "f6fa1323664f443f99f13242e07d1099"
format_stamp: "Formatted at 2026-08-06 02:55:27 on dist-test-slave-3kk6"
server_key: "51aab40be2a8f359decadf590973a638"
server_key_iv: "14a4c206630ab61fe5c285a1808e2f55"
server_key_version: "encryptionkey@0"
I20260806 02:55:27.077533 24797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:27.093614 24797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:27.097771 24797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:27.098026 24797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:27.105865 24797 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.189:38489
I20260806 02:55:27.106156 24892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.189:38489 every 8 connection(s)
I20260806 02:55:27.107128 24797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/data/info.pb
I20260806 02:55:27.107985 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24797
I20260806 02:55:27.108125 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-1/wal/instance
I20260806 02:55:27.108471 25098 external_mini_cluster.cc:1428] Setting key 7b809e21c882d973f4e0f57323598c12
I20260806 02:55:27.111061 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:27.115530 24893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.130.188@KRBTEST.COM; defaulting to no policy
I20260806 02:55:27.119511 24789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.130.188@KRBTEST.COM" created.
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:27.133736 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.119997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38992)
0806 02:55:27.120221 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.120234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.120311 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.121629 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.121633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.121840 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.121982 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.121990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.123731 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.123736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.124263 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.124271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.125170 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.125173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.125313 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.125940 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.125957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.127904 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.131181 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.131184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.131187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.131546 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.132069 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.132073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.132075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.132170 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.133085 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.133088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.133225 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.133360 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.133543 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":20}
I20260806 02:55:27.133985 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.117945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.24.130.190:36421)
0806 02:55:27.118097 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.118101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.119333 (+ 1232us) server_negotiation.cc:407] Connection header received
0806 02:55:27.120774 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.120777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.120840 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.120935 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.122604 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.123415 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.124961 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.125383 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.125506 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.129009 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.129013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.129016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.129049 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.131744 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.132409 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.132413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.132414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.132496 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.132842 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.132845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.132848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.133088 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.133184 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.133697 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.133833 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":56}
I20260806 02:55:27.134889 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.118453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36421 (local address 127.0.0.1:41772)
0806 02:55:27.119146 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.119199 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.120489 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.121121 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.121128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.121722 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.122422 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.122441 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.123569 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.123575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.124810 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.124824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.125574 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.125578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.125680 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.126220 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.126244 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.128807 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.131874 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.131888 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.131897 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.132285 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.132609 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.132612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.132614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.132725 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.133211 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.133219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.133394 (+ 175us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.133557 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.134420 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":652,"thread_start_us":212,"threads_started":1}
I20260806 02:55:27.134893 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.120199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.24.130.189:38489)
0806 02:55:27.120580 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.120590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.120629 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:55:27.121261 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.121267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.121447 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.121649 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.122175 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.123621 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.124400 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.125036 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.125202 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.128086 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.128094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.128118 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.128146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.130739 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.131717 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.131726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.131734 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.131959 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.132271 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.132275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.132277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.132952 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.133106 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.133476 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.133874 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":33280,"thread_start_us":177,"threads_started":1}
I20260806 02:55:27.136250 24893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:27.137378 24789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:27.138271 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.137981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46006)
0806 02:55:27.138123 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.138189 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:55:27.139161 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.138870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46018)
0806 02:55:27.139034 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.139093 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 02:55:27.139370 24789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:42481: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.130.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab.
Entry for principal kudu/127.24.130.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab.
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:55:27.142638 25098 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
/tmp/dist-test-taskGNnxjX/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:42481
--webserver_interface=127.24.130.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab
--principal=kudu/127.24.130.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.130.148:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.130.190:36421,127.24.130.189:38489,127.24.130.188:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:27.146021 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.136930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.24.130.189:38489)
0806 02:55:27.137037 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.137042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.137055 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:55:27.137212 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.137215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.137267 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.137355 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.137879 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.138623 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.139717 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.140051 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.140143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.142378 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.142384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.142390 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.142423 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.144272 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.144735 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.144741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.144743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.144832 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.145119 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.145122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.145123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.145322 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.145427 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.145680 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.145901 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":42}
I20260806 02:55:27.146021 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.136866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:38994)
0806 02:55:27.136984 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.137000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.137089 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.137396 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.137399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.137601 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.137739 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.137745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.138804 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.138807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.139582 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.139592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.140160 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.140162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.140244 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.140838 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.140859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.142220 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.144412 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.144415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.144416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.144625 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.144930 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.144934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.144936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.145020 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.145427 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.145431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.145525 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.145634 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.145821 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260806 02:55:27.147238 24893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:27.148052 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.147760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46022)
0806 02:55:27.147895 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.147986 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 02:55:27.148936 24824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
I20260806 02:55:27.150084 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.149864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46030)
0806 02:55:27.149979 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.150026 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:55:27.150342 24893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:42481: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
I20260806 02:55:27.170578 24789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } attempt: 1
I20260806 02:55:27.171731 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.171351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46032)
0806 02:55:27.171552 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.171628 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 02:55:27.172662 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.172420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46046)
0806 02:55:27.172539 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.172600 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:55:27.172813 24789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:42481: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
I20260806 02:55:27.199642 24893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } attempt: 1
I20260806 02:55:27.200714 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.200365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46052)
0806 02:55:27.200534 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.200614 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:55:27.201629 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.201402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46054)
0806 02:55:27.201513 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.201567 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:55:27.201787 24893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:42481: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
I20260806 02:55:27.247982 24789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } attempt: 2
I20260806 02:55:27.249027 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.248654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46064)
0806 02:55:27.248870 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.248941 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:55:27.249782 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.249554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46072)
0806 02:55:27.249682 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.249735 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 02:55:27.249907 24789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:42481: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
W20260806 02:55:27.267798 24902 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:55:27.268193 24902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:27.268258 24902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:27.268337 24902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:55:27.269987 24893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } attempt: 2
I20260806 02:55:27.271009 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.270606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46084)
0806 02:55:27.270850 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.270929 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 02:55:27.271848 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.271587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46092)
0806 02:55:27.271715 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.271770 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:55:27.272035 24893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.130.188:42481: Network error: Client connection negotiation failed: client connection to 127.24.130.188:42481: connect: Connection refused (error 111)
W20260806 02:55:27.272954 24902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:27.273012 24902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:27.273056 24902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:27.273102 24902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:27.273151 24902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:27.278822 24902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.130.148:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.130.190:36421,127.24.130.189:38489,127.24.130.188:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/krb5kdc/kudu_127.24.130.188.keytab
--principal=kudu/127.24.130.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.130.188:42481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.130.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:55:27.280040 24902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:27.281275 24902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:55:27.290468 24902 server_base.cc:1034] running on GCE node
W20260806 02:55:27.290424 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:27.290424 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:27.290452 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:27.291507 24902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:27.292805 24902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:27.294034 24902 hybrid_clock.cc:648] HybridClock initialized: now 1785984927294004 us; error 52 us; skew 500 ppm
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:27.297187 24902 init.cc:377] Logged in from keytab as kudu/127.24.130.188@KRBTEST.COM (short username kudu)
I20260806 02:55:27.298162 24902 webserver.cc:460] Webserver started at http://127.24.130.188:36629/ using document root <none> and password file <none>
I20260806 02:55:27.298806 24902 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:27.298888 24902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:27.299080 24902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:27.301280 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data/instance:
uuid: "1be86d2d802945099eae0046ec028305"
format_stamp: "Formatted at 2026-08-06 02:55:27 on dist-test-slave-3kk6"
server_key: "7ed0feda864fbc0ce75e1dcfcd361f70"
server_key_iv: "d69e77787e4a602cb59ccf31aa4ac4e8"
server_key_version: "encryptionkey@0"
I20260806 02:55:27.301857 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal/instance:
uuid: "1be86d2d802945099eae0046ec028305"
format_stamp: "Formatted at 2026-08-06 02:55:27 on dist-test-slave-3kk6"
server_key: "7ed0feda864fbc0ce75e1dcfcd361f70"
server_key_iv: "d69e77787e4a602cb59ccf31aa4ac4e8"
server_key_version: "encryptionkey@0"
I20260806 02:55:27.305923 24902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:55:27.308684 24920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:27.310065 24902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:55:27.310186 24902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal
uuid: "1be86d2d802945099eae0046ec028305"
format_stamp: "Formatted at 2026-08-06 02:55:27 on dist-test-slave-3kk6"
server_key: "7ed0feda864fbc0ce75e1dcfcd361f70"
server_key_iv: "d69e77787e4a602cb59ccf31aa4ac4e8"
server_key_version: "encryptionkey@0"
I20260806 02:55:27.310370 24902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:27.325979 24902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:27.329840 24902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:27.330088 24902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:27.337888 24902 rpc_server.cc:307] RPC server started. Bound to: 127.24.130.188:42481
I20260806 02:55:27.338233 24991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.130.188:42481 every 8 connection(s)
I20260806 02:55:27.339145 24902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/data/info.pb
I20260806 02:55:27.342264 24993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:27.342316 25098 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu as pid 24902
I20260806 02:55:27.342485 25098 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNnxjX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984624291587-25098-0/minicluster-data/master-2/wal/instance
I20260806 02:55:27.342893 25098 external_mini_cluster.cc:1428] Setting key 54fad4f0ac659626cd7437e5e71c355a
I20260806 02:55:27.347797 24993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:27.349079 24789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } attempt: 3
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.190@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:27.361172 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.344448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.24.130.188:42481)
0806 02:55:27.344934 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.344956 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.345002 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:55:27.345960 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.345987 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.346460 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.346889 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.347603 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.349107 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.350557 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.351351 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.351557 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.354932 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.354939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.354954 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.354989 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.357139 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.357708 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.357716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.357722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.357866 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.358373 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.358376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.358377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.358973 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.359198 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.359527 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.359912 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":330,"thread_start_us":134,"threads_started":1}
I20260806 02:55:27.361287 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.349604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46120)
0806 02:55:27.349782 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.349801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.349898 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.351155 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.351160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.351366 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.351482 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.351489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.352459 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.352463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.353464 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.353477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.353963 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.353966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.354056 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.354592 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.354610 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.356162 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.358891 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.358894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.358897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.359740 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.360074 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.360077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.360079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.360179 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.360660 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.360664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.360770 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.360926 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.361077 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":30}
I20260806 02:55:27.361706 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.349882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.24.130.190:36421)
0806 02:55:27.350011 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.350017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.350464 (+ 447us) server_negotiation.cc:407] Connection header received
0806 02:55:27.350687 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.350691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.350755 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.350847 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.352603 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.353107 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.354369 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.354768 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.354891 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.357862 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.357866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.357869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.357896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.359524 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.360079 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.360083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.360085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.360250 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.360636 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.360639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.360641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.360872 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.360957 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.361454 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.361567 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260806 02:55:27.361349 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.350575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.24.130.188:42481)
0806 02:55:27.350939 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.350949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.350961 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:55:27.350999 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.351001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.351034 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.351134 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.351582 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.352337 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.353605 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.353837 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.353945 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.356315 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.356324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.356326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.356351 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.358762 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.359855 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.359861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.359866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.359954 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.360279 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.360281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.360282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.360536 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.360665 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.360981 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.361111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260806 02:55:27.362272 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.349879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.190:36421 (local address 127.0.0.1:41786)
0806 02:55:27.350312 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.350368 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.350500 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.350922 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.350928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.351440 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.352488 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.352493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.353223 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.353228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.354229 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.354242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.355000 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.355003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.355135 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.355844 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.355864 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.357700 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.359642 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.359653 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.359660 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.359975 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.360451 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.360455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.360456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.360540 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.361027 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.361034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.361150 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.361890 (+ 740us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.362091 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":902,"thread_start_us":126,"threads_started":1}
I20260806 02:55:27.363234 24893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } attempt: 3
I20260806 02:55:27.364640 24993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:27.367086 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e: Bootstrap starting.
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.189@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.190@KRBTEST.COM
I20260806 02:55:27.371219 24789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e: Neither blocks nor log segments found. Creating new log.
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:55:27.372375 24789 log.cc:824] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:27.374752 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.363769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46130)
0806 02:55:27.363940 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.363956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.364051 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.364441 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.364444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.364645 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.364785 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.364796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.366021 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.366024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.366862 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.366875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.368024 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.368113 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.368674 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.368690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.370398 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.372914 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.372918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.372920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.373250 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.373620 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.373623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.373626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.373726 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.374140 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.374144 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.374254 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.374370 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.374531 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:55:27.374782 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.363802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.24.130.188:42481)
0806 02:55:27.364205 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.364209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.364221 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:55:27.364254 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.364259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.364304 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.364927 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.365018 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.365880 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.367546 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.367888 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368379 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.370555 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.370559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.370562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.370588 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.372771 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.373401 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.373407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.373411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.373510 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.373833 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.373836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.373837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.374024 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.374149 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.374426 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.374542 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":324}
I20260806 02:55:27.375175 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e: No bootstrap required, opened a new log
I20260806 02:55:27.376447 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.365370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.24.130.189:38489)
0806 02:55:27.365510 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.365516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.365641 (+ 125us) server_negotiation.cc:407] Connection header received
0806 02:55:27.365818 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.365824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.365875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.365938 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.366522 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.367382 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368401 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.368728 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368800 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.372116 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.372120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.372123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.372155 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.374485 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.375026 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.375029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.375031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.375117 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.375451 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.375454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.375456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.375737 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.375814 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.376166 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.376302 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":41}
I20260806 02:55:27.376962 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.365094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.189:38489 (local address 127.0.0.1:39008)
0806 02:55:27.365571 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.365586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.365705 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.366097 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.366099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.366294 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.366400 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.366407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.367501 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.367504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.368275 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.368285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.368962 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.369076 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.370268 (+ 1192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.370289 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.371954 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.374612 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.374615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.374617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.374902 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.375253 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.375256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.375258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.375345 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.375892 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.375894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.375984 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.376656 (+ 672us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.376805 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408}
I20260806 02:55:27.377696 24993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:27.378067 24893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: Bootstrap starting.
I20260806 02:55:27.379657 24789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.379878 24789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bfd54e97cd4023a84f49ea485dfe3e, State: Initialized, Role: FOLLOWER
I20260806 02:55:27.380215 24789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.380813 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.381045 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.381340 24893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:27.381780 24789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:27.382513 24893 log.cc:824] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.130.188@KRBTEST.COM for kudu/127.24.130.188@KRBTEST.COM
I20260806 02:55:27.385353 24893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: No bootstrap required, opened a new log
I20260806 02:55:27.386302 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.364778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.24.130.190:36421)
0806 02:55:27.364883 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.364887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.365090 (+ 203us) server_negotiation.cc:407] Connection header received
0806 02:55:27.365279 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.365282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.365331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.365382 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.366331 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.367041 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368107 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.368446 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.368513 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.372282 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.372287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.372290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.372318 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.383503 (+ 11185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.384194 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.384198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.384200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.384319 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.384686 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.384691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.384693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.384972 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.385177 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.385972 (+ 795us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.386138 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:55:27.388532 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.378389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.24.130.188:42481)
0806 02:55:27.378485 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.378489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.378502 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:55:27.378536 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.378539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.378578 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.378700 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.379187 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.379958 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.381027 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.381345 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.381422 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.384105 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.384109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.384112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.384135 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.386710 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.387188 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.387192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.387193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.387288 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.387569 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.387571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.387573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.387808 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.387880 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.388198 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.388334 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:55:27.388981 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.378234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.130.188:42481 (local address 127.0.0.1:46136)
0806 02:55:27.378336 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:27.378352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:27.378431 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:27.378762 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:27.378765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:27.378945 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:27.379043 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.379049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.380093 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.380097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.380848 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:27.380859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.381616 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.381622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:27.381696 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.382193 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:27.382216 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:27.383949 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:27.386831 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.386834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.386836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.387080 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.387386 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:27.387389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:27.387391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:27.387473 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:27.387935 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:27.387941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:27.388035 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:55:27.388676 (+ 641us) client_negotiation.cc:240] Negotiation successful
0806 02:55:27.388823 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 02:55:27.390169 24893 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.390488 24893 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fa1323664f443f99f13242e07d1099, State: Initialized, Role: FOLLOWER
I20260806 02:55:27.390925 24893 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.392285 24993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: Bootstrap starting.
I20260806 02:55:27.392539 24893 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 02:55:27 dist-test-slave-3kk6 krb5kdc[24685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.130.189@KRBTEST.COM
I20260806 02:55:27.395572 24993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:27.391202 24702 master_runner.cc:186] Error getting master registration for 127.24.130.189:38489: OK, instance_id { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" instance_seqno: 1785984927103615 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:55:27.396133 24702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:55:27.396585 24993 log.cc:824] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: Log is configured to *not* fsync() on all Append() calls
W20260806 02:55:27.398000 25018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:27.398108 25018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:27.398356 25021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.398504 25021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.400447 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.389080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.24.130.189:38489)
0806 02:55:27.389214 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.389220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.389237 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:55:27.389288 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.389293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.389341 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.389410 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.390327 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.391091 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.392228 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.392556 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.392659 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.396021 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.396025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.396028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.396060 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.397956 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.398527 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.398531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.398533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.398626 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.398986 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.398989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.398991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.399277 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.399458 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.400132 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.400277 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260806 02:55:27.400681 25037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:27.400856 25037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:27.401185 24797 master_runner.cc:186] Error getting master registration for 127.24.130.188:42481: OK, instance_id { permanent_uuid: "1be86d2d802945099eae0046ec028305" instance_seqno: 1785984927335659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:55:27.404224 24797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:55:27.407403 25098 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:27.408589 24756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "41bfd54e97cd4023a84f49ea485dfe3e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41788
I20260806 02:55:27.408859 24756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:55:27.408987 24756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:27.409103 24993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: No bootstrap required, opened a new log
I20260806 02:55:27.410142 24756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.411289 24756 leader_election.cc:288] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [CANDIDATE]: Term 1 election: Requested vote from peers f6fa1323664f443f99f13242e07d1099 (127.24.130.189:38489), 1be86d2d802945099eae0046ec028305 (127.24.130.188:42481)
I20260806 02:55:27.412395 24857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6fa1323664f443f99f13242e07d1099"
I20260806 02:55:27.412675 24857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:27.413481 24993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.413852 24993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be86d2d802945099eae0046ec028305, State: Initialized, Role: FOLLOWER
I20260806 02:55:27.414186 24857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41bfd54e97cd4023a84f49ea485dfe3e in term 1.
I20260806 02:55:27.414213 24993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.414804 24721 leader_election.cc:302] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41bfd54e97cd4023a84f49ea485dfe3e, f6fa1323664f443f99f13242e07d1099; no voters:
I20260806 02:55:27.415097 25002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:27.415375 25002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [term 1 LEADER]: Becoming Leader. State: Replica: 41bfd54e97cd4023a84f49ea485dfe3e, State: Running, Role: LEADER
I20260806 02:55:27.415710 25002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } }
I20260806 02:55:27.415820 24993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:27.417068 24958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1be86d2d802945099eae0046ec028305"
I20260806 02:55:27.417395 24958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:27.419734 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bfd54e97cd4023a84f49ea485dfe3e. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.419865 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:27.420881 24958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41bfd54e97cd4023a84f49ea485dfe3e in term 1.
I20260806 02:55:27.422161 25040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.422303 25040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.427009 25060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260806 02:55:27.427856 25062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:27.428023 25062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:27.429464 25060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:27.446332 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.418730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.24.130.189:38489)
0806 02:55:27.418857 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.418861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.432060 (+ 13199us) server_negotiation.cc:407] Connection header received
0806 02:55:27.432300 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.432304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.432364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.432419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.433570 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.434401 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.435723 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.436075 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.436157 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.441470 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.441475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.441479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.441521 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.443918 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.444514 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.444518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.444520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.444618 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.444922 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.444926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.444928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.445158 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.445301 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.445988 (+ 687us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.446139 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:55:27.452785 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.428199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.24.130.190:36421)
0806 02:55:27.428321 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.428325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.434991 (+ 6666us) server_negotiation.cc:407] Connection header received
0806 02:55:27.435182 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.435185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.435253 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.435309 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.436365 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.437104 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.442791 (+ 5687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.443186 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.443291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.448517 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.448521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.448524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.448562 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.450447 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.451024 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.451028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.451030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.451122 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.451431 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.451434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.451436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.451689 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.451810 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.452448 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.452587 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:55:27.456725 24958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [term 1 FOLLOWER]: Refusing update from remote peer 41bfd54e97cd4023a84f49ea485dfe3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:27.457535 25002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:27.461951 25040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bfd54e97cd4023a84f49ea485dfe3e. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.462106 25040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.463459 24857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [term 1 FOLLOWER]: Refusing update from remote peer 41bfd54e97cd4023a84f49ea485dfe3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:27.464098 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:27.427545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.24.130.188:42481)
0806 02:55:27.427668 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.427673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.436969 (+ 9296us) server_negotiation.cc:407] Connection header received
0806 02:55:27.438358 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.438363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.438420 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.438473 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.439384 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.440189 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.441976 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:27.442327 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:27.442414 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:27.449680 (+ 7266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:27.449687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:27.449693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:27.449735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:27.451547 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.452096 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.452100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.452102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.452192 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:27.462799 (+ 10607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:27.462805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:27.462807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:27.463134 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:27.463313 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:27.463761 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 02:55:27.463932 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:55:27.464452 25002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:27.468899 25021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bfd54e97cd4023a84f49ea485dfe3e. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.469056 25021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.475715 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.475911 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:27.476133 25021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.476409 25021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.476900 25069 mvcc.cc:205] Tried to move back new op lower bound from 7315394262830895104 to 7315394262698696704. Current Snapshot: MvccSnapshot[applied={T|T < 7315394262830895104}]
I20260806 02:55:27.479328 25072 mvcc.cc:205] Tried to move back new op lower bound from 7315394262830895104 to 7315394262698696704. Current Snapshot: MvccSnapshot[applied={T|T < 7315394262830895104}]
I20260806 02:55:27.480162 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.480293 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:27.481609 25040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41bfd54e97cd4023a84f49ea485dfe3e" member_type: VOTER last_known_addr { host: "127.24.130.190" port: 36421 } } peers { permanent_uuid: "f6fa1323664f443f99f13242e07d1099" member_type: VOTER last_known_addr { host: "127.24.130.189" port: 38489 } } peers { permanent_uuid: "1be86d2d802945099eae0046ec028305" member_type: VOTER last_known_addr { host: "127.24.130.188" port: 42481 } } }
I20260806 02:55:27.481747 25040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:27.484870 25070 mvcc.cc:205] Tried to move back new op lower bound from 7315394262830895104 to 7315394262698696704. Current Snapshot: MvccSnapshot[applied={T|T < 7315394262830895104}]
I20260806 02:55:27.485113 25060 catalog_manager.cc:1347] Generated new cluster ID: 472178ad44784ddda30bc04cc2398dd6
I20260806 02:55:27.485193 25060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:27.516765 25060 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:27.518395 25060 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:27.534126 25060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41bfd54e97cd4023a84f49ea485dfe3e: Generated new TSK 0
I20260806 02:55:27.535223 25060 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:55:27.567634 24998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.564672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.24.130.188:42481)
0806 02:55:27.564833 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.564838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.566380 (+ 1542us) server_negotiation.cc:407] Connection header received
0806 02:55:27.566598 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.566601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.566704 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.566762 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.567494 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46160: BlockingRecv error: recv got EOF from 127.0.0.1:46160 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 02:55:27.569181 24794 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.565663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.24.130.190:36421)
0806 02:55:27.565801 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.565805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.568188 (+ 2383us) server_negotiation.cc:407] Connection header received
0806 02:55:27.568371 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.568374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.568426 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.568476 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.569049 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41798: BlockingRecv error: recv got EOF from 127.0.0.1:41798 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:55:27.571168 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.569844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.24.130.189:38489)
0806 02:55:27.569964 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.569968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.570265 (+ 297us) server_negotiation.cc:407] Connection header received
0806 02:55:27.570384 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.570387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.570440 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.570526 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.571041 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39026: BlockingRecv error: recv got EOF from 127.0.0.1:39026 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 02:55:27.575701 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.573871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.24.130.189:38489)
0806 02:55:27.573980 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.573985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.574536 (+ 551us) server_negotiation.cc:407] Connection header received
0806 02:55:27.574866 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.574869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.574950 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.575007 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.575577 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39040: BlockingRecv error: recv got EOF from 127.0.0.1:39040 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:55:27.581445 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.579690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.24.130.189:38489)
0806 02:55:27.579803 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.579808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.580366 (+ 558us) server_negotiation.cc:407] Connection header received
0806 02:55:27.580571 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.580574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.580633 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.580703 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.581329 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39046: BlockingRecv error: recv got EOF from 127.0.0.1:39046 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 02:55:27.589205 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.587221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.24.130.189:38489)
0806 02:55:27.587355 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.587360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.588131 (+ 771us) server_negotiation.cc:407] Connection header received
0806 02:55:27.588360 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.588366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.588425 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.588480 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.589082 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39056: BlockingRecv error: recv got EOF from 127.0.0.1:39056 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:55:27.598444 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.596325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.24.130.189:38489)
0806 02:55:27.596497 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.596502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.597228 (+ 726us) server_negotiation.cc:407] Connection header received
0806 02:55:27.597465 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.597469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.597536 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.597605 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.598264 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39058: BlockingRecv error: recv got EOF from 127.0.0.1:39058 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 02:55:27.612006 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.609908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.24.130.189:38489)
0806 02:55:27.610070 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.610075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.610787 (+ 712us) server_negotiation.cc:407] Connection header received
0806 02:55:27.611051 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.611055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.611120 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.611183 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.611873 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39062: BlockingRecv error: recv got EOF from 127.0.0.1:39062 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:55:27.633329 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.631414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.24.130.189:38489)
0806 02:55:27.631571 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.631578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.632118 (+ 540us) server_negotiation.cc:407] Connection header received
0806 02:55:27.632368 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.632374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.632451 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.632515 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.633199 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39064: BlockingRecv error: recv got EOF from 127.0.0.1:39064 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:55:27.669926 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.668727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.24.130.189:38489)
0806 02:55:27.668932 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.668936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.668957 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:55:27.669119 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.669122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.669172 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.669255 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.669786 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39072: BlockingRecv error: recv got EOF from 127.0.0.1:39072 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260806 02:55:27.737706 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.736434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.24.130.189:38489)
0806 02:55:27.736583 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.736587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.736852 (+ 265us) server_negotiation.cc:407] Connection header received
0806 02:55:27.736938 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.736941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.736997 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.737086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.737594 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39086: BlockingRecv error: recv got EOF from 127.0.0.1:39086 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:55:27.869436 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:27.868249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.24.130.189:38489)
0806 02:55:27.868450 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:27.868454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:27.868538 (+ 84us) server_negotiation.cc:407] Connection header received
0806 02:55:27.868672 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:27.868676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:27.868731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:27.868813 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:27.869294 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39098: BlockingRecv error: recv got EOF from 127.0.0.1:39098 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 02:55:28.129438 24898 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:55:28.128105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.24.130.189:38489)
0806 02:55:28.128254 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:28.128258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:28.128400 (+ 142us) server_negotiation.cc:407] Connection header received
0806 02:55:28.128601 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:28.128604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:28.128662 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:28.128761 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:28.129310 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57556: BlockingRecv error: recv got EOF from 127.0.0.1:57556 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:55:28.402274 25037 catalog_manager.cc:1259] Loaded cluster ID: 472178ad44784ddda30bc04cc2398dd6
I20260806 02:55:28.402420 25037 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: loading cluster ID for follower catalog manager: success
I20260806 02:55:28.404192 25037 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: acquiring CA information for follower catalog manager: success
I20260806 02:55:28.405467 25037 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6fa1323664f443f99f13242e07d1099: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:55:28.429530 25062 catalog_manager.cc:1259] Loaded cluster ID: 472178ad44784ddda30bc04cc2398dd6
I20260806 02:55:28.429641 25062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: loading cluster ID for follower catalog manager: success
I20260806 02:55:28.431344 25062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: acquiring CA information for follower catalog manager: success
I20260806 02:55:28.432618 25062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1be86d2d802945099eae0046ec028305: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:55:28.647926 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:28.643912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.24.130.189:38489)
0806 02:55:28.644255 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:28.644259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:28.644277 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:55:28.644340 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:28.644342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:28.644501 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:28.644611 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:28.644873 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.645446 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.646288 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.646523 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.646621 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:28.646983 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:28.647440 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:28.647539 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:28.647598 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 02:55:28.647651 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260806 02:55:28.655537 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:28.651441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.24.130.188:42481)
0806 02:55:28.651693 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:28.651696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:28.651730 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:55:28.651898 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:28.651900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:28.652038 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:28.652148 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:28.652369 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.652895 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.653729 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.654006 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.654118 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:28.654590 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:28.655163 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:28.655269 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:28.655346 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 02:55:28.655404 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260806 02:55:28.663132 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:28.658836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.24.130.189:38489)
0806 02:55:28.659042 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:28.659047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:28.659432 (+ 385us) server_negotiation.cc:407] Connection header received
0806 02:55:28.659549 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:28.659552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:28.659606 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:28.659693 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:28.659959 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.660578 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.661745 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.662094 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.662196 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:28.662518 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:28.662597 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:28.662693 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:28.662868 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 02:55:28.662948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:55:28.663623 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:28.658904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.24.130.188:42481)
0806 02:55:28.659166 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:28.659170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:28.659467 (+ 297us) server_negotiation.cc:407] Connection header received
0806 02:55:28.659652 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:28.659658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:28.659718 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:28.659806 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:28.660132 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.660919 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.662046 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.662359 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.662462 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:28.662829 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:28.662914 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:28.662985 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:28.663406 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 02:55:28.663472 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260806 02:55:28.663899 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:28.658789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.24.130.190:36421)
0806 02:55:28.659338 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:28.659342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:28.659355 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:55:28.659480 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:28.659483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:28.659671 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:28.659806 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:28.660193 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.661013 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.661911 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:28.662282 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:28.662401 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:28.662807 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:28.663183 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:28.663301 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:28.663661 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 02:55:28.663737 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":78,"threads_started":1}
I20260806 02:55:28.672241 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24702
I20260806 02:55:28.690181 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24797
I20260806 02:55:28.706318 25098 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNnxjX/build/debug/bin/kudu with pid 24902
2026-08-06T02:55:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3438 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6828 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (304413 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST